Raking in the dough on Free and Open Source Software

I’m writing this on the third day after the “Heartbleed” bug in OpenSSL devasted internet security, and while I have been very critical of the OpenSSL source code since I first saw it, I have nothing but admiration for the OpenSSL crew and their effort.

In particular considering what they’re paid for it.

Inspired by an article in Wall Street Journal which tangentially touches on the lack of funding for OpenSSL development, I have decided to write up my own experiences with funding Open Source Software development in some detail.

I’ve been in the software industry for 30 years now, and I have made a living more or less directly from Open Source Software for the most recent 15 years.

Sometimes the money came from helping a customer use Open Source Software, some times I wrote the Open Source Software for their needs and sometimes, as with the Varnish Moral License I get paid to develop and maintain Open Source Software for the greater common good.

FreeBSD community funding

My first crowd-funding of Free and Open Source Software, was in 2004, where I solicited the FreeBSD community for money, so that I could devote three to six months of my time to the FreeBSD disk-I/O subsystem.

At that time I had spent 10 years as one of the central and key FreeBSD developers, so there were no question about my ability or suitability for the task at hand.

But in 2004 crowd-funding was not yet “in”, and I had to figure out how to do it myself.

My parents brought me up to think that finances is a private matter but I concluded that the only way you could ask strangers to throw money at you, would be to run an open book, where they could see what happened to them, so I did open books.

My next dilemma was about my rate, again I had always perceived my rate to be a private matter between me and my customers.

My rate is about half of what most people expect – because I wont work for most people: I only work on things I really care about.

One of my worries therefore were that publishing my rate would undercut friends and colleagues in the FreeBSD project who made a living consulting.

But again, there were no way around it, so I published my rate but made every attempt to distinguish it from a consulting rate, and I never heard any complaints.

And so, having agonized over the exact text and sounded it off on a couple of close friends in the FreeBSD project, I threw the proposal out there – and wondered what would happen next.

I had a perfectly safe fall-back plan, you have to when you have two kids and a mortgage to feed, but I really had no idea what would happen.

Worst case, I’d cause the mother of all bikesheds get thrown out of the FreeBSD project, and be denounced for my “ideological impurity” with respect to Free and Open Source Software.

Best case, I expected to get maybe one or two months funded.

The FreeBSD community responded overwhelmingly, my company has never sent as many invoices as it did in 2004, and my accountant nearly blew a fuse.

And suddenly I found myself in a situation I had never even considered how to handle: How to stop people from sending me money.

I had simply set up a PayPal account, (more on that in a bit), and at least at that time, there were no way to prevent people from dropping money into it, no matter how much you wanted to stop them.

In the end I managed to yell loud enough and only got overfunded a few percent, and I believe that my attempt to deflect the surplus to the FreeBSD Foundation gave them a little boost that year.

So about PayPal: The first thing they did was to shut my account, and demand all kinds of papers to be faxed to them, including a copy of my passport, despite the fact that Danish law was quite clear on that being illegal. Then, as now, their dispute resolution process was less than user-friendly, and in the end it took an appeal to a high-ranking officer in PayPal and quite a bit of time to actually get the money people had donated.

I swore to myself that next time, if there ever came a next time, PayPal would not be involved. Besides, I found their fees quite excessive.

In total I made EUR27K, and it kept my kids fed and my bank happy for the six months I worked on it.

And work I did.

I’ve never had a harsher boss than those six months, and it surprised me how much it stressed me, because I felt like I was working on a stage, with the entire FreeBSD project in audience, wondering if I were going to deliver the goods or not.

As a result, the 187 donors certainly got their moneys worth, most of that half year I worked 80 hour weeks, which made me decide not to continue, despite many donors indicating that they were perfectly willing to fund several more months.

Varnish community funding

Five years later, having developed Varnish 1.0 for Norways “Verdens Gang” newspaper, I decided to give community funding a go again.

Wiser from experience, I structured the Varnish Moral License to tackle the issues which had caused me grief the first time around:

Contact first, then send money, not the other way around, and also a focus on fewer larger sponsors, rather than people sending me EUR10 or USD15 or even, in one case, the EUR1 which happened to linger in his PayPal Account.

I ran even more open books this time, on the VML webpages you can see how many hours and a one-line description of what I did in them, for every single day I’ve been working under the VML since 2010.

I also decided to be honest with myself and my donors, one hour of work was one hour of work – nobody would benefit from me dying from stress.

In practice it doesn’t quite work like that, there are plenty of thinking in the shower, emails and IRC answers at all hours of the day and a lot of “just checking a detail” that happens off the clock, because I like my job, and nothing could stop me anyway.

In each of 2010, 2011 and 2013 I worked around 950 hours work on Varnish, funded by the community.

In 2012 I only worked 589 hours, because I was building a prototype computer cluster to do adaptive optics real-time calculations for the ESO Extremely Large Telescope (“ELT”) – There was no way I could say no to that contract :-)

In 2014 I actually have hours available do even more Varnish work, and I have done so in the ramp up to the 4.0.0 release, but despite my not so subtle hints, the current outlook is still only for 800 hours to be funded, but I’m crossing my fingers that more sponsors will appear now that V4 is released. (Nudge, nudge, wink, wink, he said knowingly! :-)

Why Free and Open Source costs money

Varnish is about 90.000 lines of code, the VML brings in about EUR90K a year, and that means that Varnish has me working and caring about issues big and small.

Not that I am satisfied with our level of effort, we should have much better documentation, our wish-list of features is far too long and we take too long to close tickets.

But I’m not going to complain, because the Heartbleed vulnerability revealed that even though OpenSSL is about three to five times larger in terms of code, the OpenSSL Foundation Inc. took in only about EUR700K last year.

And most of that EUR700K was for consulting and certification, not for “free-range” development and maintenance of the OpenSSL source code base so badly needs.

I really hope that the Heartbleed vulnerability helps bring home the message to other communities, that Free and Open Source Software does not materialize out of empty space, it is written by people.

People who love what we do, which is why I’m sitting here, way past midnight on a Friday evening, writing this pamphlet.

But software is written by people, real people with kids, cars, mortgages, leaky roofs, sick pets, infirm parents and all other kinds of perfectly normal worries of an adult human being.

The best way to improve the quality of Free and Open Source Software, is to make it possible for these people to spend time on it.

They need time to review submissions carefully, time to write and run test-cases, time to respond and fix to bug-reports, time to code and most of all, time to think about the code.

But it would not even be close to morally defensible to ask these people to forego time to play with their kids, so that they instead develop and maintain the software that drives other peoples companies.

The right way to go – the moral way to go – and by far the most productive way to go, is to pay the developers so they can make the software they love their living.

How to fund Free and Open Source Software

One way is to hire them, with the understanding that they spend some company time on the software.

Experience has shown that these people almost invariably have highly desirable brains which employers love to throw at all sorts of interesting problems, which tends to erode the “donated” company time.

But a lot of Free and Open Source Software has been, and still is developed and maintained this way, with or without written agreements or even knowledge of this being the case.

Another way is for software projects to set up foundations to collect money and hire developers. This is a relatively complex thing to do, and it will only be available for larger projects.

The Apache Foundation “adopts” smaller projects inside their field of interest, and I believe that works OK, but I’m not sure if it can easily be transplanted to different topics.

The final way is to simply throw money a the developers, the way the FreeBSD and Varnish communities have done with me.

It is a far more flexible solution with respect to level of engagement, national boundaries etc. etc, but in many ways it demands more from both sides of the deal, in particular with respect to paperwork, taxes and so on.

Conclusion

I am obviously biased, I derive a large fraction of my relatively modest income from community funding, for which I am the Varnish community deeply grateful.

But biased as I may be, I believe that the Varnish community and I has shown that a tiny investment goes a long way in Free and Open Source Software.

I hope to see that mutual benefit spread to other communities and projects, not just to OpenSSL and not just because they found a really bad bug the other day, but to any community around any piece of software which does serious work for serious companies.

Thanks in advance,

Poul-Henning, 2014-04-11