Grant Proposal: Improving Perl 5
Sun, 21-Aug-2011 by
Karen Pauley
edit post
<![CDATA[
Nicholas Clark has submitted a grant proposal under our new "Perl 5 Core Maintenance Fund":http://www.perlfoundation.org/perl_5_core_maintenance_fund.
Before we vote on this proposal we would like to get feedback and endorsements from the Perl community. Please leave feedback in the comments or send email with your comments to karen at perlfoundation.org.
**Name**: Dr. Nicholas Clark
**Contact Information**:
IRC: nicholas
Email: nick@ccl4.org
**Project Title**: Improving Perl 5
**Synopsis**:
Free up one of the Perl 5 core's key contributors to work non-stop on making Perl 5 better.
**Benefits to Perl 5 Core Maintenance**:
Perl 5 is an amazingly flexible language, and in the 16 years since Larry Wall released 5.000, the Perl interpreter has been modified in ways no-one could have foreseen, and ported to platforms that didn't even exist then. Not unsurprisingly, the Perl interpreter's 16 year old codebase has suffered in the process, and is not as clean or tidy as it could be. This makes it harder for people to learn how the internals work, getting in the way of bringing new blood into the project. For people experienced in the internals, this makes it harder to find and fix bugs, and to add desired new functionality. Funding work to reduce the technical debt of the internals will increase the efficiency of the existing maintainers, and make it easier to recruit and train new maintainers in the future.
**Deliverable Elements**:
I propose to adopt the same model as Dave's successful "ongoing grant":http://news.perlfoundation.org/2010/02/grant-proposal-fixing-perl5-co.html.
Like his grant, there are intentionally no pre-defined deliverables for this project. I intend to devote around 400 hours over the next 3 months to work on improving the core, paid by the hour at the same below-commercial rate as Dave. Like him, I would post a weekly summary on the p5p mailing list detailing activity for the week, allowing the grant managers and active core developers to verify that the claimed hours tally with actual activity, and thus allow early flagging of any concerns. Likewise, missing two weekly reports in a row without prior notice would be grounds for one of my grant managers to terminate this grant.
Exactly as Dave does, once per calendar month I would claim an amount equal to $50 x hours worked. I would issue a report similar to the weekly ones, but summarising the whole month. The report would need to be signed off by one of the grant managers before I get paid. Note that this means I am paid entirely in arrears.
At the time of my final claim, I would also produce a report summarising the activity across the whole project period.
Also, (the "nuclear option"), the grant managers would be allowed, at any time, to inform the board that in their opinion the project is failing, and that the TPF board may then, after allowing me to present my side of things, to decide whether to terminate the project at that point (i.e. to not pay me for any hours worked after I was first informed that a manager had "raised the alarm").
I view this grant as a proof of concept - if it goes well for everyone involved, I expect to apply to extend it. Hence the specific tasks mentioned below are given as examples of the things I'd like to get done over a longer period, *not* a list things I expect to get done in three months.
**Project Details**:
I think that the work that I would do to improve Perl 5 would mostly fall into one of 4 main classes: refactoring, bug fixing, helping other contributors, and adding features - with refactoring the most prominent and adding features the least. "No plan survives contact with the enemy" is often the case when it comes to working with the Perl 5 core - the proper fix for a problem usually turns out to involve a lot of collateral bug fixing or other improvements. Hence I can't give a meaningful list of "what I will work on" because it will rapidly become so outdated as to be completely wrong. Instead, I have an illustrative list of example tasks, and metrics to judge success:
1: Refactor the existing implementation (both build time and run time), without adding or removing features.
For all of these, suitable metrics to judge success are reducing any of:
bq. a: amount of code (without increasing code complexity - golfing is cheating)
b: size of compiled code (executables and shared objects)
c: run time memory use
d: run time CPU use
For example I currently have ideas to investigate on how to:
bq. a: further reduce the size of various structures such as symbol tables
b: remove duplication between headers, dump.c and Perl code
c: reduce the size of BOOT code generated by ExtUtils::ParseXS
d: reduce copy-paste repetition in the build system
e: reduce complexity and duplication of header macros by using static inline functions
2: Fixing bugs, much like Dave is doing. Given a moderate amount of communication about which general area each is working in, I don't think that there's much risk of duplicate work here. (And that would probably scale to a third person, before a proper manager would be needed)
However, fixing both "big" and "small" bugs would be useful. Big, because Dave will welcome the help; small, because it is good to get the total bug count down. In particular, I'd likely be able to look into problems that the release manager deems to be "showstoppers" for a stable release.
I'd like to review every use of I32 in the core, as I suspect that most are wrong - my hunch is that at least 90% should be one of U32, IV or STRLEN. I'd also like to review the entire PerlIO implementation. I don't expect to start on either of these tasks in this initial grant.
Bug fixing can be tracked by RT tickets being successfully resolved. Code review is harder, although it could be tracked by the number of RT tickets *opened*.
3: Ensuring that threads on perl5-porters don't stall, and questions don't go unanswered.
Often volunteer contributors, or would-be contributors, mail the list with questions or part finished work, stalled until their requests are answered. Some messages are easy for anyone answer quickly, so someone answers. Some messages are easy for particular people with the right knowledge to answer, so the message gets answered. But some messages aren't easy for anyone to answer quickly, so there is no reply, and a potential contribution or even contributor is lost.
It's not just newcomers or people round the periphery of Perl 5 development - several of the active committers write e-mails that request help, or that ought to generate discussion, but no-one has the time to think through the problem, so no-one replies. For similar requests where I have responded, to write my message has often taken about an hour of digging and prep-work, in order to write the reply that the original deserved. If this sort of work isn't covered by the grant, then it simply isn't going to get done, as it makes no economic sense to work on something hard and unfunded when something equally hard, funded and potentially more fun is also available.
This is *hard* to measure quantitatively, but I think that it's important to somehow cater for it. I wouldn't expect it to average out much over 10% of a week's work, or peak higher than 20%. The reporting structure means that it can be measured qualitatively - the grant managers and active core developers can compare the figure claimed with replies in the mailing list archives, and request clarification, deny part-payment, or even refer the grant to the board for termination, if appropriate.
4: Actually add features
The "success criteria" for new features are that
bq. a: other people think that the plan is good
b: that the plan gets implemented
Currently the only features that I think would be beneficial to add are all at the internals level. Specifically
bq. a: implementing and benchmarking Chip's "view" support at the API level
b: vtables for array and hash access, to permit
i) decoupling the "restricted hash" implementation from the SV flags
ii) removing the tied array/tied hash code from the common case
iii) a better implementation of sparse arrays
iv) flexibility for typed arrays and hashes as CPAN modules
Whilst working, I would also keep in mind how we might develop process and structure to keep things working well, if future funding would enable us to cover the costs of more than 2 or 3 people working on core maintenance.
For the grant, I'm specifically not proposing to
1: be a release manager. I think that current rotating volunteer system that Jesse put in place is working well.
2: managing the synchronisation of CPAN modules and interaction with their authors - what sane organisation benefits from promoting its best programmers into management positions?
3: act as language designer - I don't feel that I'm good at this.
4: take sole or principal responsibility for tasks that anyone could do, such as reviewing or applying general patches sent to the list.
**Project Schedule**:
I expect that I can deliver 400 hours of work in approximately 3 months. As described above in "deliverable elements", specific tasks in "project details" are examples of the types of things that I intend to work on. I certainly can't get all done in the first 400 hours, and it may turn out that other tasks are better use of TPF's money, in the opinion of myself, my grant managers and the active Perl 5 developers.
I am available to start work on this project immediately.
**Bio.**:
Of the top 3 contributors to Perl 5, I'm the only one still active, and the only one who has removed more lines of code than they added. I've shipped more stable releases of Perl than anyone else in the past 15 years.
As well as that, over the past 13 years I've
bq. * Reduced the memory usage of most core data structures
* Reduced the runtime CPU usage by about 5%
* Replaced platform specific code with unified build tools
* Restructured extensions to a layout which is much simpler to maintain
* Devised and implemented the changes needed to be able to cleanly move modules from core to CPAN, and thereby removed Switch
* And along the way fixed many bugs.
**Endorsed by**: Jesse Vincent, Tim Bunce, Marcus Holland-Moritz
**Amount Requested**: $20,000
**Suggestions for Grant Manager**: Marcus Holland-Moritz, Ricardo Signes.
]]>
Comments (21)
Please, please, please!
Nick knows what he is doing. He has been at it for ages, and not only has brought his own original work to perl, but has helped improve the work of dozens of others.
perl would benefit enormously from having more funded work to sort out problems that won't get done in contributors' leisure time, and Nick is obviously one of the top people we could possibly fund.
For me, this is a no-brainer.
Please accept this proposal.
Yes, this sounds great. Let's do it.
I am very happy to see the Perl 5 core get a little love, not to mention some money.
I think that Nick is a perfect candidate for a grant of this type: he brings both his experience and his enthusiasm.
First, obviously he has worked extensively on the core and is one of the most qualified people for this type of work.
Then he brings his enthusiasm, which to me makes it likely that first he will enjoy this work and do it brilliantly, and second that should the grant one day stop, he would keep on working on the core as he has done before, and Perl 5 wouldn't loose a contributor.
So yes to the grant.
Yes, yes and yes - Nick should be given a grant.
as a relative newcomer when it comes to trying to fix things in core I welcome all work that is done to make it simpler to understand. If this also makes it faster, smaller and more awesome in general it's an added bonus
Therefor this gets my +Inf (or a close enough approximation) support.
Just do it!
You won't find a better candidate than Nicholas.
Please vote "yes" for this proposal, Nicholas is one of the few very well qualified core hackers for this task.
Also seems like a no-brainer to me. Someone who knows the internals as well as he does who wants to be paid to work on them and make all of Perl better for everyone else? Yes please.
Yes. I have great confidence that Nicholas, like Dave Mitchell, will fulfill the terms of his grant to the great benefit of the whole Perl community.
This is really exciting and awesome. Couldn't find a better person!
Please vote "yes" on this.
Sounds good!
Please accept the grant proposal!
Nicholas has always been a valuable contributor to the core. Saying "yes" seems like a no-brainer to me.
That's exactly the kind of work Perl needs to keep it competitive in the long run. Go Go Go! :)
This seems to be an eminently sensible idea that no self-respecting Perl developer could disagree with.
+1
Slap the golden handcuffs on 'im before he changes his mind!
Yes,
I met Nick in person last year, and speaking as someone with no particular interest in the interpreter itself, I was impressed with his ability to explain its inner workings clearly to a relatively un-engaged person such as myself.
Wow.
So I'm all ready to endorse this grant but I see that I'm probably going to be the last in a very long list. Maybe I should vote "no" just to change it up .... Nah! :-)
As has been said before, this is a no-brainer: please accept this proposal.
Funny, it hasn't been a *week* since the proposal was published here, and I already feel like we're wasting time talking about it :)
This is a really important issue, and even people not deep inside core activity such as myself are pretty confident that Nick is the right person for the job.
Please accept this proposal!
I use perl in the modern world, so do vendors who love linux.
