Recently in Perl 5 Development Category

Dave Mitchell has requested an extension of $20,000 for his Maintaining the Perl 5 Core grant. He also has requested an hourly rate change from $50 to $60. This will allow him to dedicate another 333 hours to this work.

During this grant he sent regular reports to the p5p mailing list as well as providing monthly summary reports that have been published on this site, the most recent of which are linked below:

Before we make a decision on this extension, we would like to have a period of community consultation. Please leave feedback in the comments field below or if you prefer, send email with your comments to makoto at perlfoundation.org.

If successful this extension will be funded from the Perl 5 Core Maintenance Fund.

This is a monthly report by Dave Mitchell on his grant under Perl 5 Core Maintenance Fund. We thank the TPF sponsors to make this grant possible.

I spent May mainly:

* heavily reworking perl's internal sprintf implementation.
  I've now pushed a smoking branch containing about 100 commits which
  fixes bugs, audits the code for possible integer overflows, makes the
  code simpler and simpler to understand, and improves performance by
  around 15%.

* Investigating short-string PVs. I've pushed a proof-of-concept branch
  that allows short strings (< 16 bytes) to be stored directly in the
  body of an SV. It also opens up a pathway to improve the COW
  implementation, which I intend to work on at some point.

* helping to write and proofread the perldelta.pod file for the 5.26.0
  release.

SUMMARY:
     96:22 RT #131260 sprintf implementation
      0:58 RT #131296 Time-HiRes/Makefile.PL @INC issue
     18:47 investigate short-string PVs
     16:25 perldelta fixups
      4:34 process p5p mailbox
    ------
    137:06 TOTAL (HH::MM)


 189.4 weeks
2728.9 total hours
  14.4 average hours per week

There are 71 hours left on the grant

TPF Board has received a new grant application under Perl 5 Core Maintenance Fund.

Before we vote on this proposal, we would like to get feedback from the Perl community. Please leave feedback in the comments or, if you prefer, email your comments to makoto at perlfoundation.org.

full name: Andrew Main

IRC nickname: Zefram

project title: Zefram maintaining the Perl 5 core

synopsis

I'd like a grant to work on the Perl 5 core, concentrating on tricky and obscure issues of the core internals. The grant would extend to the full range of a core committer's activities, along the lines of my past work on the core. The grant would allow me to put much more time into the Perl 5 core than I have been able to recently.

benefits to Perl 5 core maintenance

this grant would overcome the present limitation on the amount of time I can devote to Perl, namely my need for an income. The core would benefit from the increased application of my rare knowledge and skills, including my deep understanding of much of the core internals. It would especially benefit through the fixing of troublesome bugs, and more generally through the addressing of issues that require in-depth familiarity with the internals.

deliverable elements

Like the similar grants for Dave Mitchell, Nicholas Clark, and Tony Cook, I do not propose specific technical deliverables for this project. The details of what work I tackle would be led by the exigencies of bug reports and the direction of discussions among core developers (on the public mailing list, the IRC channel, and at times on the closed security report mailing list). The main deliverable, then, is time spent on core maintenance work, at a rate of USD 50 per hour.

Secondary deliverables are the required reports, weekly to p5p and monthly to TPF. These will list the issues worked on. The grant manager representing p5p can see how this relates to my code commits and mailing list activity, providing an opportunity to raise any concerns.

project details

This project covers the full range of core maintenance activity that I have performed in the past. Within this mix, I would favour activity of which the fewest people are capable: work that takes advantage of my particular knowledge of core internals and my particular abilities. In general, the highest priority work would be the diagnosis and resolution of bugs that appear to involve obscure internals. (Note that diagnosis can radically change the appearance of a bug in this respect.)

Activities covered by this grant would include, but are not limited to:

  • diagnosing reported bugs
  • fixing bugs, whether reported or not
  • discussing core issues on the p5p mailing list
  • writing and improving documentation
  • code refactoring
  • release engineering
  • implementation of new core features, where approved through discussion

Specifically excluded from this project scope is work on unapproved new features, because of the risk of (perception of) abuse. I do not seek a license to arbitrarily perform major speculative work on TPF's dime. But where a new feature has been discussed prior to implementation, and there is the appropriate consensus that it should be implemented, then the endeavour is no longer an individual's speculation, and that implementation work can fall within the scope of this project. Looking back at features I have added in the past, I developed call-checker and parser plugins without discussion, so they would not have been in scope for this type of grant. Subroutine signatures, on the other hand, I developed in response to a fairly detailed consensus reached on the IRC channel, and with the pumpking's specific blessing, so that would have been in scope.

project schedule

This project covers 400 hours of deliverable work, which I anticipate delivering at a rate averaging somewhere between 20 and 30 hours per week, thus taking somewhere between three and five months. I am not committing to any steady rate of work; extensive variance of time worked from week to week is to be expected. I can start as soon as approved.

If the arrangement of this grant turns out to be satisfactory, I would likely seek an extension of it on similar terms.

biography

I'm a freelance computer programmer. I have been using Perl since the year 2000, and C since 1990. I've been contributing to CPAN since 2004, now having 76 distributions there, many of them XS language extensions. I've been contributing code to the Perl 5 core since 2007. My core development work has included diagnosis and fixing of daunting bugs, design and implementation of new subsystems (such as parser plugins and subroutine signatures), and much discourse on the mailing list.

endorsed by: Sawyer X, Matthew Horsfall, Dagfinn Ilmari Mannsaker

amount requested: USD 20_000

suggestions for grant manager: Sawyer X has agreed to serve as manager.

This is a monthly report by Dave Mitchell on his grant under Perl 5 Core Maintenance Fund. We thank the TPF sponsors to make this grant possible.

The main things I did last month were:

Firstly, fixing various issues with scopes in regexes. In particular,
(RT #126697), code blocks sometimes failed to undo localisations when
backtracking. For example the $s below wasn't always being restored when
the B part of the match failed and it backtracked to try another A - where
A represents something complex like (\w+\s+)* which can match in multiple
ways:

    /A(?{ local $s = ...})B/

As part of that work, non-greedy matching of complex sub-expressions with
captures and repeated backtracking was made more efficient under some
circumstances; for example the following now runs about 25% faster:

    $s = ("a" x 1000);
    $s =~ /^(?:(.)(.))*?[XY]/ for 1..10_000;

Secondly, improving 't/TEST -deparse'.

The -deparse option to t/TEST causes it to run all the core's test
scripts, but after running them through the deparser first. Many of these
modified scripts are currently known to fail, and there is an exclusion
file, Porting/deparse-skips.txt, which is supposed to list the known
failures. However, over time, new failures have appeared which are not in
the exclusion list. Last August I did some work on Deparse.pm and managed
to reduce some of the expected and unexpected failures, but since then
more failures have crept in.

My recent work includes: modifying t/TEST so that it distinguishes
between expected failures and unexpected passes, and warning of unknown
files in Porting/deparse-skips.txt; purging Porting/deparse-skips.txt to
account for files that have been renamed or are no longer in the core, and
to reflect the current state of things; and fixing Deparse.pm to:
    * better handle 'BEGIN { use_ok() }';
    * better handle 'BEGIN { require expr }' (as opposed to require Bareword);
    * deparse lexical var attributes, e.g. 'my $foo :bar(baz)';
    * avoid a 'deep recursion' warning;
    * handle an escaped literal tab char in a pattern, e.g
        /.....\ ..../x where the whitespace char following the backslash is
        actually a tab; previously the deparse failed to emit the backslash;
    * handle declarations with multiple lexical vars within a pattern code
      block, e.g. /(?{ my ($x, $y) = @a; })/

Because we're currently in code freeze, this as been pushed as
smoke-me/davem/deparse and will be merged in 5.27.x.

Thirdly, reviewing and fixing tickets in the security queue. There's quite
a lot of tickets in the security queue due to fuzzing, where if the fuzzer
detects a use-after-free or buffer overrun for example, the reporter
submits it to the security queue rather than the normal queue. Once
examined, 95% of the time it will be found to be harmless or
non-exploitable, but until someone has assessed and fixed it, it lingers
as an open security ticket.

Perl 5 Core Hackathon (p5hack) Report

perl5-hack-logo-04.jpg

(Graphics credit to Leonardo Maia.)

Thanks to The Perl Foundation and our beloved sponsors, a large portion of the critical contributors to the Perl 5 Porters (p5p), the core group of Perl 5 developers, were able to get together on November 11th, 2016 at the Booking.com headquarters in Amsterdam, The Netherlands, for our very first Perl 5 Core hackathon, nicknamed Perl 5 Hack.

This is our report.


It was a four-day event, composed of discussions (centralized and specialized) on a myriad of important topics, knowledge sharing, and cooperative hacking. It provided us with the ability to cut down several months worth of discussions and loss of time due to timezone differences.

We worked on several topics, such as the Configure phase, core modules, security updates to Perl library loading order ("dot in @INC€), improving compilation under C++11, Core API and internals, replacements for the buggy utf8 IO layer, conditional lexical variable warnings, and deparsing some internal warnings information (${^WARNING_BITS}).

This alone would have been quite the fit, but we also achieved a few things along with those discussions. Some of our achievements include:

  • Cleaning up dead hints code (pre-ELF)
  • Improved our release guide
  • Improved our Configure process
    • Removing hundreds of complicated C conditionals
    • Faster Configure script
  • Cleaned up our install output
  • Improved our API documentation
  • Massive speed improvements to core modules
  • Explored the new Test2 testing framework for Perl
  • Reviewed and listed all current deprecations
  • Tied all deprecations to an exact version, to improve transparency and communicate better our intentions regarding deprecated features and syntax
  • Cleaned up core code:
    • Removed obscure and unused SVs
    • Removed dead code
    • Moved Unicode::Normalize to upstream blead
  • Had done optimizations, such as when assigning to references.
  • Reduced memory usage in core modules: Data::Dumper and Time::Local.
  • Fuzzed core modules
  • Have rewritten much of our readline implementation
  • Introduced the indented HEREDOC feature, being released in the upcoming 5.26

We raised a lot of topics and we made numerous important and valuable changes to the core code and core utilities, but we have also held in-depth discussions on the following topics:

  • Our COW implementation and where to take it in the future
  • Efficiency of our hash bucket sizing
  • Overloading indication
  • Our policy on editing dual-life modules
  • OP tree cloning
  • Subroutine Signatures and their introspection
  • The internals of the Perl to C compiler and what may be gleaned from it on improving perl
  • The contribution hurdles when working with RT
  • Vulnerability triage, classification, and handling - covering our security handling policies
  • EBCDIC-portability
  • Unicode (and specifically grapheme delimiters)
  • Regular Expression internals
  • Vtable-based hashes

Most of these have led to changes in policy, improvements or cleanups in core, and a "leg-up" on additional research into those topics.

All in all, despite it only being the first hackathon, it had already proved successful and beneficial. For this reason we intend to holding future hackathons, while both preparing better for them, as well as communicating better our work on them onward to the community.

Thank you to all attendees: Aaron Crane, Abigail, Brian Fraser, Dagfinn Mannsåker, Dave Mitchell, Dennis Kaarsemaker, Gonzalo Diethelm, H. Merijn Brand, John Lightsey, J. Nick Koston. Karl Williamson, Leon Timmermans, Lukas Mai, Matthew Horsfall, Mattia Barbon, Nicholas Clark, Nicolas Rochelemagne, Sawyer X, Steffen Mueller, Stevan Little, Todd Rinaldo, Viekntiy Fusenov, and Yves Orton,

And thank you to The Perl Foundation for helping in organizing this event, and to all of our sponsors for contributing and assisting in this event: Our Diamond Sponsors, Booking.com; our Platinum Sponsors of cPanel and Craigslist; Silver Sponsors, Bluehost; and Bronze Sponsors of Assurant, Grant Street Group and Signature Information Solutions, and to FastMail, for their support of this, and future Perl Foundation activities.

A special thank you to the Perl community for the support it has given us and to the Perl 5 Porters group.

Sawyer X and Todd Rinaldo, Perl 5 Hackathon organizers.

Dave Mitchell writes:

I spent December:

1) looking for quick some wins on speeding up perl compile-time, using
'perl -MCPAN -e1' as a typical example of loading and compiling several
.pm files.  I tweaked Perl_yyparse() and shaved ~2% off the compile time;
then tweaked Perl_sv_gets() and shaved another ~2% off, and also got a
pleasing run-time boost for line reading, with reading a big list of words
now ~8% faster on my system:

    perl -e'$i++ while (<>)' /usr/share/dict/words

2) investigating possible ways of making the regex engine faster by (in
some cases) processing 8 bytes at a time on 64-bit platforms. This
culminated in an this email thread:

    http://nntp.perl.org/group/perl.perl5.porters/241891

3) as usual, working on a few miscellaneous tickets.

SUMMARY:
      1:30 RT #130385 Bleadperl breaks DNS-LDNS
      1:00 RT #130385 Bleadperl breaks List-Pairwise
      1:00 RT #130398 Bleadperl breaks Method-Signatures
      4:26 [perl #129199] Difficult-to-trigger panic
      1:01 [perl #130247] Perl_rpeep(OP *): Assertion `oldop' failed
      0:55 [perl #130307] Bug: Regex matches when it shouldn't
      0:33 [perl #130311] heap-buffer-overflow Perl_yyparse
      0:55 fix build warnings and smoke failures
      1:40 fix some CPAN modules under PERL_OP_PARENT
     19:00 investigate making find_by_class faster()
      2:37 optimise Perl_sv_gets()
      4:45 optimise Perl_yyparse()
     12:20 process p5p mailbox
      3:22 review RT tickets
      3:04 review security tickets
    ------
     58:08 TOTAL (HH::MM)

 170.1 weeks
2338.8 total hours
  13.7 average hours per week

There are 61 hours left on the grant

Dave Mitchell writes:

I spent October mainly working on two things.

First, I optimised some common arithmetic operators: + - *, so that for the very common case of both args being simple ints in ranges that won't overflow, or both being floats, a simple C-level + or whatever can be directly done. For more complex or mixed args, it falls back to the existing slower code. For ++ and --, I optimised the simple integer case. I also improved the core SET[iun], PUSH[iun] and XPUSH[iun] macros, which set the pad targ to a numeric value and push it on the stack. Since PADTMPs and lex vars (which is typically what a pad targ is) are often used in the same way, e.g. always assigned and used as an integer, those macros now check whether the targ is already of the right type, and if so directly set the value, rather than just blindly calling sv_setiv() etc. The combination of the above makes the nbody benchmark (lots of floating-point vector arithmetic) about a third faster.

Second, I continued overhauling perl's context stack and dynamic scope implementation.

Summary

59:16 #124156: death during unwinding causes crash
0:09 [perl #117341] av_undef's POD is confusing
0:33 [perl #125937] 'x' operator on list causes segfault with possible stack corruption
4:35 [perl #126082] unshift to @ISA
1:11 [perl #126145] Problem with stack moving fix for Perl_load_module
2:23 [perl #126170] Assertion failed: S_finalize_op (op.c:2562)
1:55 [perl #126229] POSIX::strerror() clears $!
2:27 [perl #126309] 'x' operator on list causes segfault and confuses valgrind, 64-bit version
0:29 [perl #126472] Bleadperl v5.23.3-33-g6768377 breaks HANENKAMP/Tie-Simple-1.03.tar.gz
0:32 add perldelta entries
13:00 make arithmetic faster
0:34 optimise the Boyer-Moore string finder (as used in REs and index())
11:45 process p5p mailbox

98:49 Total (HH::MM)

As of 2015/10/28: since the beginning of the grant:

106.4 weeks
1600.0 total hours
15.0 average hours per week

There are 0 hours left on the grant.

About TPF

The Perl Foundation - supporting the Perl community since 2000. Find out more at www.perlfoundation.org.

About this Archive

This page is an archive of recent entries in the Perl 5 Development category.

Parrot development is the previous category.

Perl 6 Development is the next category.

Find recent content on the main index or look in the archives to find all content.

Pages

OpenID accepted here Learn more about OpenID
Powered by Movable Type 6.2.2