Nicholas Clark has submitted a grant proposal under our new Perl 5 Core Maintenance Fund.
Before we vote on this proposal we would like to get feedback and endorsements from the Perl community. Please leave feedback in the comments or send email with your comments to karen at perlfoundation.org.
Name: Dr. Nicholas Clark
Project Title: Improving Perl 5
Free up one of the Perl 5 core's key contributors to work non-stop on making Perl 5 better.
Benefits to Perl 5 Core Maintenance:
Perl 5 is an amazingly flexible language, and in the 16 years since Larry Wall released 5.000, the Perl interpreter has been modified in ways no-one could have foreseen, and ported to platforms that didn't even exist then. Not unsurprisingly, the Perl interpreter's 16 year old codebase has suffered in the process, and is not as clean or tidy as it could be. This makes it harder for people to learn how the internals work, getting in the way of bringing new blood into the project. For people experienced in the internals, this makes it harder to find and fix bugs, and to add desired new functionality. Funding work to reduce the technical debt of the internals will increase the efficiency of the existing maintainers, and make it easier to recruit and train new maintainers in the future.
I propose to adopt the same model as Dave's successful ongoing grant.
Like his grant, there are intentionally no pre-defined deliverables for this project. I intend to devote around 400 hours over the next 3 months to work on improving the core, paid by the hour at the same below-commercial rate as Dave. Like him, I would post a weekly summary on the p5p mailing list detailing activity for the week, allowing the grant managers and active core developers to verify that the claimed hours tally with actual activity, and thus allow early flagging of any concerns. Likewise, missing two weekly reports in a row without prior notice would be grounds for one of my grant managers to terminate this grant.
Exactly as Dave does, once per calendar month I would claim an amount equal to $50 x hours worked. I would issue a report similar to the weekly ones, but summarising the whole month. The report would need to be signed off by one of the grant managers before I get paid. Note that this means I am paid entirely in arrears.
At the time of my final claim, I would also produce a report summarising the activity across the whole project period.
Also, (the "nuclear option"), the grant managers would be allowed, at any time, to inform the board that in their opinion the project is failing, and that the TPF board may then, after allowing me to present my side of things, to decide whether to terminate the project at that point (i.e. to not pay me for any hours worked after I was first informed that a manager had "raised the alarm").
I view this grant as a proof of concept - if it goes well for everyone involved, I expect to apply to extend it. Hence the specific tasks mentioned below are given as examples of the things I'd like to get done over a longer period, not a list things I expect to get done in three months.
I think that the work that I would do to improve Perl 5 would mostly fall into one of 4 main classes: refactoring, bug fixing, helping other contributors, and adding features - with refactoring the most prominent and adding features the least. "No plan survives contact with the enemy" is often the case when it comes to working with the Perl 5 core - the proper fix for a problem usually turns out to involve a lot of collateral bug fixing or other improvements. Hence I can't give a meaningful list of "what I will work on" because it will rapidly become so outdated as to be completely wrong. Instead, I have an illustrative list of example tasks, and metrics to judge success:
1: Refactor the existing implementation (both build time and run time), without adding or removing features.
For all of these, suitable metrics to judge success are reducing any of:
a: amount of code (without increasing code complexity - golfing is cheating)
b: size of compiled code (executables and shared objects)
c: run time memory use
d: run time CPU use
For example I currently have ideas to investigate on how to:
a: further reduce the size of various structures such as symbol tables
b: remove duplication between headers, dump.c and Perl code
c: reduce the size of BOOT code generated by ExtUtils::ParseXS
d: reduce copy-paste repetition in the build system
e: reduce complexity and duplication of header macros by using static inline functions
2: Fixing bugs, much like Dave is doing. Given a moderate amount of communication about which general area each is working in, I don't think that there's much risk of duplicate work here. (And that would probably scale to a third person, before a proper manager would be needed)
However, fixing both "big" and "small" bugs would be useful. Big, because Dave will welcome the help; small, because it is good to get the total bug count down. In particular, I'd likely be able to look into problems that the release manager deems to be "showstoppers" for a stable release.
I'd like to review every use of I32 in the core, as I suspect that most are wrong - my hunch is that at least 90% should be one of U32, IV or STRLEN. I'd also like to review the entire PerlIO implementation. I don't expect to start on either of these tasks in this initial grant.
Bug fixing can be tracked by RT tickets being successfully resolved. Code review is harder, although it could be tracked by the number of RT tickets opened.
3: Ensuring that threads on perl5-porters don't stall, and questions don't go unanswered.
Often volunteer contributors, or would-be contributors, mail the list with questions or part finished work, stalled until their requests are answered. Some messages are easy for anyone answer quickly, so someone answers. Some messages are easy for particular people with the right knowledge to answer, so the message gets answered. But some messages aren't easy for anyone to answer quickly, so there is no reply, and a potential contribution or even contributor is lost.
It's not just newcomers or people round the periphery of Perl 5 development - several of the active committers write e-mails that request help, or that ought to generate discussion, but no-one has the time to think through the problem, so no-one replies. For similar requests where I have responded, to write my message has often taken about an hour of digging and prep-work, in order to write the reply that the original deserved. If this sort of work isn't covered by the grant, then it simply isn't going to get done, as it makes no economic sense to work on something hard and unfunded when something equally hard, funded and potentially more fun is also available.
This is hard to measure quantitatively, but I think that it's important to somehow cater for it. I wouldn't expect it to average out much over 10% of a week's work, or peak higher than 20%. The reporting structure means that it can be measured qualitatively - the grant managers and active core developers can compare the figure claimed with replies in the mailing list archives, and request clarification, deny part-payment, or even refer the grant to the board for termination, if appropriate.
4: Actually add features
The "success criteria" for new features are that
a: other people think that the plan is good
b: that the plan gets implemented
Currently the only features that I think would be beneficial to add are all at the internals level. Specifically
a: implementing and benchmarking Chip's "view" support at the API level
b: vtables for array and hash access, to permit
i) decoupling the "restricted hash" implementation from the SV flags
ii) removing the tied array/tied hash code from the common case
iii) a better implementation of sparse arrays
iv) flexibility for typed arrays and hashes as CPAN modules
Whilst working, I would also keep in mind how we might develop process and structure to keep things working well, if future funding would enable us to cover the costs of more than 2 or 3 people working on core maintenance.
For the grant, I'm specifically not proposing to
1: be a release manager. I think that current rotating volunteer system that Jesse put in place is working well.
2: managing the synchronisation of CPAN modules and interaction with their authors - what sane organisation benefits from promoting its best programmers into management positions?
3: act as language designer - I don't feel that I'm good at this.
4: take sole or principal responsibility for tasks that anyone could do, such as reviewing or applying general patches sent to the list.
I expect that I can deliver 400 hours of work in approximately 3 months. As described above in "deliverable elements", specific tasks in "project details" are examples of the types of things that I intend to work on. I certainly can't get all done in the first 400 hours, and it may turn out that other tasks are better use of TPF's money, in the opinion of myself, my grant managers and the active Perl 5 developers.
I am available to start work on this project immediately.
Of the top 3 contributors to Perl 5, I'm the only one still active, and the only one who has removed more lines of code than they added. I've shipped more stable releases of Perl than anyone else in the past 15 years.
As well as that, over the past 13 years I've
* Reduced the memory usage of most core data structures
* Reduced the runtime CPU usage by about 5%
* Replaced platform specific code with unified build tools
* Restructured extensions to a layout which is much simpler to maintain
* Devised and implemented the changes needed to be able to cleanly move modules from core to CPAN, and thereby removed Switch
* And along the way fixed many bugs.
Endorsed by: Jesse Vincent, Tim Bunce, Marcus Holland-Moritz
Amount Requested: $20,000
Suggestions for Grant Manager: Marcus Holland-Moritz, Ricardo Signes.