Grant Proposal: Fixing Perl5 Core Bugs

| 14 Comments

David Mitchell has submitted a grant proposal, which if accepted would make use of a portion of the funding generously provided to TPF by Booking.com.

Before the Board votes on this proposal we would like to get feedback and endorsements from the Perl community. Please leave feedback in the comments or send email with your comments to karen at perlfoundation.org.

Grant Title: Fixing perl5 core bugs

Name: David Mitchell

Amount Requested: $25,000

Synopsis

Recently, booking.com donated $50K for the "further development and
maintenance of the Perl programming language". I would like part of that
money to to be used to fund me for approximately six months to devote 50%
of my time fixing "hard" core perl5 bugs.

Benefits to the Perl Community

There are currently approximately 1200 open and 300 new bug reports in the
perl5 bug queue. Although some of these are of the "5.003_08 does not
build on platform X" variety, many are current: for example, almost 500 of
them were created after the release of 5.10.0. As the perl core has become
more and more gnarly, and the pool of experienced but active core hackers
has declined, these bugs are just piling up and not getting fixed,
especially the hard ones. With this funding, I would would be able to
devote serious time and effort to making a dent in this queue.

Note that unlike many large open source projects, perl has no paid
developers devoted to bug fixing.

Deliverables

Unusually for a TPF grant, there are not clear-cut deliverables for this
project. I intend to devote 500 hours of my time over the next six months
fixing perl core bugs. The net result will be a list of bug numbers that
have been diagnosed, and (hopefully) fixed. Because it's impossible to
predict in advance how difficult a bug is going to be to diagnose and fix
(or indeed whether it is even fixable), I can't commit in advance to a
fixed list of bugs that I will fix over the course of the grant. Nor is it
realistic to have a bounty per fixed bug; I would end up not getting
rewarded for time spent on difficult bugs, and conversely I would have a
strong incentive to cherry-pick easy bugs, defeating the purpose of the
grant.

Therefore, monitoring of my progress will become important (see below).

Project Details

I think this has been fully covered above.

Inch-stones

Note that due to the length and scale of this project, it is suggested
that there be two project managers, who can spread the monitoring load
between them as they see fit.

Since this project is heavily based on hours worked and the monitoring
thereof, I would post a weekly summary on the p5p mailing list which
details, for each bug worked on that week, how many hours were spent on
diagnosis and fixes, plus any bug status changes. This frequent feedback
would allow the grant managers and active core developers (who will be
aware of any recent commits and other activity of mine) to observe whether
my claimed hours bear any relation to actual activity and results, and
thus allow early flagging of any concerns.

Missing two weekly reports in a row without prior notice would be grounds
for terminating the project.

Once per calendar month I would claim an amount equal to $50 x hours worked.
I would issue a report similar to the weekly ones, but summarizing the
whole month. The report would need to be signed off by one of the
project managers before I get paid. Note that this means I am paid
entirely in arrears.

At the time of my final claim, I would also produce a report summarising
the activity across the whole project period.

Also, (the "nuclear option"), I suggest that either of the project managers
be allowed, at any time, to inform the board that in their opinion the
project is failing badly, and that the TPF board may then, after allowing
me to present my side of things, to vote whether to terminate the project
at that point (i.e. to not pay me for any hours worked after I was first
informed that a manager had "raised the alarm").

To ensure that at there are at least some visible results for the hours
spent, I would be required have closed at least one bug per 20 hours
before being able to claim money for those hours. (I would hope to close
more bugs than that, but by setting a low baseline, I'm not tying my
hands, while still allowing TPF to have something visible for publicity
purposes during the interim.)

Project Schedule

I am available to start work on this project immediately.

The project is expected to take six months. I am self-employed, which
allows me a good deal of flexibility. By promising approximately 50% of my
time, this gives me the ability to continue with my existing commitments
to other clients, while deferring seeking new clients. As such, the weekly
hours I devote to perl are likely to be highly variable, but hopefully
averaging out to about 20 hours per week. If for some reason I find that I
have spent less than 500 hours at the end of the six months, then I will
continue the project until until the 500 hours been spent, with the
proviso that that the TPF board are free to terminate the project at any
time after the six months. Conversely, if I manage to devote more than 20
hours per week, then my monthly payments will be accordingly larger, and
the project will terminate early (once the 500 hours are spent).

Note that it is currently my intention that the after six months I will
apply for a further $25K extension, although there is no obligation for me
to do so, nor for TPF to approve it.

Bio

I'm a freelance UNIX sysadmin and programmer living in the UK. I have been
using perl since 1993, and have been fixing core perl 5 bugs since 2001.
I have had commit rights since 2003 and I was pumpking for the 5.10.1 perl
release.

In short, I am one of only a handful of active people who understand large
parts of the perl internals and who can thus fix "hard" bugs.

14 Comments

I like the way this grant proposal is organized, particularly how it builds in accountability.

If people like Nicholas Clark and other core contributors were to endorse this, I think this would be a no-brainer for funding.

I'm not really familiar with a lot that p5p does, nor am I qualified to comment on whether or not these 1500 bugs are serious or not. But this seems like a great idea to me. Having a number that large of bugs isn't good for Perl's reputation of stability and progress. And it's hard to find volunteers to do bug triage (which I assume a lot of this work will be).

This grant is for a lot of money, but at the same time there are a lot of precautions built in to make sure that money won't be payed out without accompanying work. And it would be nice to have someone being payed to work on Perl's core even if it is part time.

I'm acquainted with the quality of Dave's patches, and being a long-standing perl5 committee, I fully support this grant.

Generally, I like the idea and I like the strict accountability framework that Dave M. has designed. I think the gnarly bugs pile up because fixing them takes concentrated time and this would provide for an expert to spend that concentrated time consistently for several months.

I would like to see a few things added to this grant:

  1. More emphasis on bug triage and prioritization of effort
  2. Support for closing of irrelevant tickets
  3. More connection to the existing meta-ticket "process"

On the first point -- we've built up huge technical debt allowing so many bug reports to pile up that it's hard to know where Dave's time is best spent. I would support him spending the first month doing nothing but triage on the roughly 500 "high" or "medium" severity tickets to determine (a) if they correctly classified, (b) whether they can be closed as fixed/end-of-life and (c) a rough sense of difficulty. I'd then like to see the remaining time spent on the higher severity tickets across a mix of difficulties (banging out some easier ones and diving into some trickier ones).

On the second point -- I believe closing irrelevant or non-replicable tickets is also valuable work. I wouldn't want to see that become the bulk of Dave's effort, but I wouldn't want him to avoid doing so for lack of "credit". I would expect the managers to consider such closings as counting towards "quota" while providing some degree of oversight (above and beyond p5p members) that such closings are appropriate.

On the third point -- there were a number of meta-tickets created to track blocking, non-blocking, "needs expert assessment" and "end-of-life-wontfix" issues. I'd like to see Dave M. work within (or improve) that framework as he goes and to focus particularly on the "needs expert assessment" meta-ticket. This would provide means for community members to flag particular tickets for Dave to review, giving them a voice in guiding the effort.

I have confidence in David's ability and commitment. I fully support this proposal.

I fully support this proposal.

I fully support this proposal as well. Dave is one of the few people with the ability and experience to address many of the hard core perl bugs. This grant will give him some time to do so in a useful, concentrated manner. His work on perl-5.10.1 is more than ample evidence that he is capable of understanding difficult core issues, and has a good sense of the appropriate balance between maintaining backwards compatibility and the occasional necessity to break things in order to move forward.

1. On triage: It is certainly likely that a considerable amount of time will effectively be devoted to triaging efforts, but I think it would be ill-advised for TPF to require too much specifically in that area. Bug reports vary greatly in how easy they are to triage. Especially for the hard problems that Dave proposes to tackle, the time and effort required to do steps (a), (b), and (c) is such that it often wouldn't make sense to stop there and simply report it as a "hard" problem. Instead, that's the precise best time to try to solve it. Saying "triage it in month 1, but don't fix it until month 2" might involve re-learning everything that was learned in month 1 about the bug.

More broadly, however, I completely agree -- as Dave goes through the bug database, he should leave plenty of notes and comments on tickets if he has anything useful to add.

2. & 3.: I agree with both of these points. Indeed, I'd hope the meta-ticket process would evolve in useful ways as a result of this process.

I approve of this proposal. I note that proposal is specifically about "hard"
core bugs, not directly about managing the bug queue. I agree with this.

Historically we have at times had various unsung (volunteer) heroes quietly
dig through it. (I'm aware of Bram, Steve Peters and Schwern, and doubtless
unaware of others.) Usually it's possible to knock it back to about 1300 open
bugs, but rarely further, because those that remain are too hard to be fixed
in an a volunteer's available time.

Hence why we need to pay someone to break out of this loop.

I like the proposed mechanisms for ensuring accountability. To my mind they
strike a good compromise between verifying that work is being done, whilst not
attempting to micromanage or dictate which bugs should be investigated.
Specifically I like the plan for payment by hours worked, rather than bugs
closed, because it will make for better quality fixes. The opposite would mean
a subconscious bias towards the quickest fix for the symptoms, in order to
close the bug, rather than investigating and fixing the root cause. The
opposite would also lead to more bugs being closed, but simpler bugs,
rather than the harder bugs that would likely never otherwise be addressed.


I understand where David Golden is coming from in his suggestion, but I do not
agree with it formally becoming part of the proposal.

Specifically, I object to the idea of spending the first month solely triaging
tickets as

  1. it undermines the checks and balances built into the proposal
  2. doing this exclusively will be thoroughly demoralising for Dave
  3. there will be no early visible progress
  4. it interferes with fixing high priority hard bugs for 5.12.0, 5.12.1 etc.

Having done a grant for TPF working on the Perl core, and having worked alone
on Ponie, I'm well aware of the pressures that one is under. Specifically, it
can be very lonely at times if you're stuck on a bug, with no-one to bounce
ideas off of help you. (This happens on the hard bugs. And if you're the only
non-volunteer, you can't expect anyone else to drop everything to have a
brainstorming session with you.) There is also the demotivational danger of
thinking that everything is being left to you, because you're the paid dogsbody,
and they're all volunteers. We need to guard against that too, else this
work won't be sustainable long term.

Hence why I'm also wary of having a formal requirement to for the "community"
to be able to flag up tickets for Dave to be obliged to look at. At one time
at Fotango there was what was dubbed "Management by RT"*. Developers were
assigned work by having tickets dumped into their queue, without any real form
of discussion about what, or why it was important. Hence I think it's
important that guidance on what could be done is a two way process, and
I think that it's important that this is done by the grant managers and the
pumpking and release engineers, rather than anyone and everyone. You wouldn't
put your third line support people on the helpdesk telephones, so why would
you do the equivalent to Dave?

[* note, RT is not at fault here. It's management by ticketing system, where
the ticketing system in use happened to be RT. For some reason RT is popular
as a ticketing system. I can't think why :-)]

The planets don't align like this very often. TPF has a cash drawer bulging a bit larger than usual due to the generous booking.com donation, requiring an ambitious grant of significant and lasting benefit to Perl. Many good things are happening in Perl 5 core development, but the proposal is unfortunately quite accurate about the size and nature of the bug queue, and the tendency of very part-time volunteers to punt on the really hard ones. Jesse is demonstrating all sorts of creative leadership but quite explicitly said when he took up the 5.12 patch pumpkin, "I am not a perlguts rockstar."[1]

Dave is. Obviously. The exact thing that is hardest to find, that is most urgently needed, and that you can't put an ad in the paper for even if you have a lot more money than TPF does is exactly what Dave is offering at exactly the price TPF can afford. The proposal shows an excellent balance of simplicity, flexibility, autonomy, accountability, and a "start yesterday" mentality.

Please accept the proposal as is and allow discussions of other things Dave might do to go on in parallel with getting some bugs fixed; they can always become part of what I hope will be a renewed grant 500 hours later.


[1] http://www.nntp.perl.org/group/perl.perl5.porters/2009/10/msg152913.html

Responding to Nicholas' points:

Given the size of the grant, I think it's appropriate to have an early read on where Dave is planning to spend his time. That's why I suggested triaging the high/medium severity bugs (~500). If Dave thinks that's too much time just on triage, that's fine. I'd still encourage triage of just the high severity bugs. My point is that I want to see a plan of attack to ensure that his efforts are directed towards the most important areas. I support "up to" (not mandating) a month of triage/planning as valuable thinking work. I'm not saying he should only triage during that time. I'm saying that we should be willing to pay for a thoughtful approach, which I expect will take a few weeks to figure out.

On the "expert assessment" meta tickets -- again, I don't suggest this be mandated for the reasons you suggest. But then again, I think the list of people who can attach tickets to a meta ticket is limited so you already have a first filter. (I could be wrong.) My suggestion is that he triage in a similar way, using those tickets and that he try to assess the higher severity tickets in the "expert assessment" queue. I think that it's just as frustrating for the community to do bug triage and then have no experts assess the ones that need it. The meta-ticket system broadly seems to be working well for Perl 5.11.X efforts, so reinforcing what's working seems like a good idea. It shouldn't consume his efforts, but I think it's a valuable area for at least some of his time and attention.

I support this proposal, and I don't think there should be more formalities than proposed - Dave should be able to concentrate on hacking as much as possible.

I support this proposal as-is. I don't think Dave will have any trouble finding relevant bugs to work on, especially once he starts sending reports to perl5-porters which can give him good ones to work on in the very unlikely case that he can't find any.

I fully support this proposal as is. I've known Dave through his work on the Perl core for almost six years now. I fully trust that Dave will continue working much as he has during that time by tackling the hard and higher priority bugs.

Leave a comment

About this Entry

This page contains a single entry by Karen published on February 7, 2010 3:39 PM.

Find recent content on the main index or look in the archives to find all content.