February 2017 Archives

This is a monthly report by Tony Cook on his grant under Perl 5 Core Maintenance Fund. We thank the TPF sponsors to make this grant possible.

Approximately 38 tickets were reviewed, and 8 patches were

[Hours]         [Activity]
 13.82          #122490 (sec) more merge conflicts
                #122490 (sec) more merge conflicts, track down warning
                #122490 (sec) track down warning sources, start merging
                test changes
                #122490 (sec) more test merging, testing, debugging
                #122490 (sec) debugging
                #122490 (sec) debugging
  0.97          #126228 build testing, apply to blead
 16.08          #127663 testing, apply hash seed env suppression patch,
                back to in-place changes
                #127663 work on chdir test, testing, debugging, make
                mg_freeext() API and fix docs
                #127663 cleanup, threads handling, threads test
                #127663 more threads testing, try to make it fail with
                #127663 more try to make it fail with fork and succeed,
                work on fix, code polish
                #127663 hoist some work back up, testing
                #127663 uncombine thread/fork child handling which I
                combined by accident, work on more tests and find a couple
                of cleanup issues
                #127663 more tests
                #127663 post patch to ticket
  0.22          #128528 (sec) review and comment
  0.88          #128998 track down when it was fixed, ticket management
  0.30          #129012 make public, comment and close
  1.88          #129024 review, make public, check fix backports to 5.24,
                non-trivial backport to 5.22, comment
  1.30          #129125 check, testing, apply to blead
  1.65          #129149 apply patch, test #130557 case, testing, make
                public apply to blead, comment on #130557
  0.08          #129187 check and merge into #129149
  0.95          #129190 rebase with some conflicts, testing, make public,
                apply to blead
  0.17          #129199 make public, comment and close
  2.62          #129274 (sec) try to find an alternative attack
                #129274 more trying to break it, write regression test,
                testing, make public, apply to blead
  2.12          #129292 review code, debugging, make public and comment
  1.77          #129298 review patches, research, consider whether changes
                improve perl
                #129298 more consideration, ask khw
  4.32          #129340 (sec) review code, think about solutions
                #129340 (sec) work on a solution, testing
                #129340 (sec) write a regression test, testing
                #129340 (sec) suggested changes, testing
                #129340 (sec) research, comment with updated patch
  0.50          #129342 (sec) test provided patch, create a test case and
  0.45          #129377 (sec) review patch, look for similar issues,
  1.32          #129840 (sec) review, testing
                #129840 get it to test, merge into 129377
  0.40          #129848 review and make public
  1.53          #129861 (sec) debugging
  0.42          #129887 (sec) review and comment
  0.82          #129963 research, make public and link to stack-not-
                refcounted meta ticket
  0.92          #129975 debugging, make public and link to stack-not-
                refcounted meta ticket
  0.28          #130100 make public and point at list discussion on
  0.73          #130256 debugging, make public and link to stack-not-
                refcounted meta ticket
  1.67          #130262 apply patch with noise, test #130558 case,
                testing, make public, push to blead, comment on #130558
  0.18          #130321 (sec) debugging
  0.68          #130504 review, testing, apply to blead
  0.43          #130560 comment
  0.90          #130567 reproduce, suspect 94749a5ed was bad, ask khw on
                #130567 irc discussion
  1.35          #130569 (sec) comment
  2.85          #130578 debugging
                #130578 debugging, comment
  0.58          #130591 review discussion and comment
  0.33          #130614 research and comment
  1.57          #130635 review changes, check memory use, testing, comment
                #130635 comment
  1.55          #130675 debugging, #p5p discussion
                #130675 debugging, #p5p comment, ticket comment on #130679
  0.42          comment on deprecations thread
 69.01 hours total

Perl 5 Core Hackathon (p5hack) Report


(Graphics credit to Leonardo Maia.)

Thanks to The Perl Foundation and our beloved sponsors, a large portion of the critical contributors to the Perl 5 Porters (p5p), the core group of Perl 5 developers, were able to get together on November 11th, 2016 at the Booking.com headquarters in Amsterdam, The Netherlands, for our very first Perl 5 Core hackathon, nicknamed Perl 5 Hack.

This is our report.

It was a four-day event, composed of discussions (centralized and specialized) on a myriad of important topics, knowledge sharing, and cooperative hacking. It provided us with the ability to cut down several months worth of discussions and loss of time due to timezone differences.

We worked on several topics, such as the Configure phase, core modules, security updates to Perl library loading order ("dot in @INC€), improving compilation under C++11, Core API and internals, replacements for the buggy utf8 IO layer, conditional lexical variable warnings, and deparsing some internal warnings information (${^WARNING_BITS}).

This alone would have been quite the fit, but we also achieved a few things along with those discussions. Some of our achievements include:

  • Cleaning up dead hints code (pre-ELF)
  • Improved our release guide
  • Improved our Configure process
    • Removing hundreds of complicated C conditionals
    • Faster Configure script
  • Cleaned up our install output
  • Improved our API documentation
  • Massive speed improvements to core modules
  • Explored the new Test2 testing framework for Perl
  • Reviewed and listed all current deprecations
  • Tied all deprecations to an exact version, to improve transparency and communicate better our intentions regarding deprecated features and syntax
  • Cleaned up core code:
    • Removed obscure and unused SVs
    • Removed dead code
    • Moved Unicode::Normalize to upstream blead
  • Had done optimizations, such as when assigning to references.
  • Reduced memory usage in core modules: Data::Dumper and Time::Local.
  • Fuzzed core modules
  • Have rewritten much of our readline implementation
  • Introduced the indented HEREDOC feature, being released in the upcoming 5.26

We raised a lot of topics and we made numerous important and valuable changes to the core code and core utilities, but we have also held in-depth discussions on the following topics:

  • Our COW implementation and where to take it in the future
  • Efficiency of our hash bucket sizing
  • Overloading indication
  • Our policy on editing dual-life modules
  • OP tree cloning
  • Subroutine Signatures and their introspection
  • The internals of the Perl to C compiler and what may be gleaned from it on improving perl
  • The contribution hurdles when working with RT
  • Vulnerability triage, classification, and handling - covering our security handling policies
  • EBCDIC-portability
  • Unicode (and specifically grapheme delimiters)
  • Regular Expression internals
  • Vtable-based hashes

Most of these have led to changes in policy, improvements or cleanups in core, and a "leg-up" on additional research into those topics.

All in all, despite it only being the first hackathon, it had already proved successful and beneficial. For this reason we intend to holding future hackathons, while both preparing better for them, as well as communicating better our work on them onward to the community.

Thank you to all attendees: Aaron Crane, Abigail, Brian Fraser, Dagfinn Mannsåker, Dave Mitchell, Dennis Kaarsemaker, Gonzalo Diethelm, H. Merijn Brand, John Lightsey, J. Nick Koston. Karl Williamson, Leon Timmermans, Lukas Mai, Matthew Horsfall, Mattia Barbon, Nicholas Clark, Nicolas Rochelemagne, Sawyer X, Steffen Mueller, Stevan Little, Todd Rinaldo, Viekntiy Fusenov, and Yves Orton,

And thank you to The Perl Foundation for helping in organizing this event, and to all of our sponsors for contributing and assisting in this event: Our Diamond Sponsors, Booking.com; our Platinum Sponsors of cPanel and Craigslist; Silver Sponsors, Bluehost; and Bronze Sponsors of Assurant, Grant Street Group and Signature Information Solutions, and to FastMail, for their support of this, and future Perl Foundation activities.

A special thank you to the Perl community for the support it has given us and to the Perl 5 Porters group.

Sawyer X and Todd Rinaldo, Perl 5 Hackathon organizers.

Dave Mitchell writes:

There are two main things I did in December (along with the usual
day-to-day stuff).

First I sorted out infrastructure for ops which are called in boolean

Currently PADHV and RV2HV ops are flagged specially if they are not only
used in scalar context, but where the result they return will only
ever be used as a boolean value; for example 'if (%h) {...}'

This was originally done since a hash in scalar context returned a bucket
usage ratio, which became expensive to calculate when we no longer
stored the bucket occupancy count with each hash.

In 5.25.x we've changed it so that in scalar context it now only returns a
key count, which is always known and doesn't need to be calculated on the
fly. But it can still be quicker to return just a boolean value in
these cases rather than an integer.

My work took this special-case boolean context detection code in the
peephole optimiser, and improved it so that:

 * it fixes a bunch of bugs - e.g. if (!%h) {...} wasn't recognised as
 * it expands the number of situations that are recognised as boolean
   context, such as grep:  e.g. grep %$_, @AoH;
 * it adds a reasonably comprehensive set of tests for checking that ops
   have been flagged as boolean under various boolean scenarios;
 * it allows other ops to be easily added to the boolean detection in the
   optimiser; and to the test suite.

I've also tweaked RV2HV so that in the negative case it just returns
&PL_sv_no rather than sv_2mortal(newSViv(0)))

I've also pushed a branch davem/boolref2 that allows the ref() function to
be optimised in boolean context; so 'if (ref $r) {...}' doesn't have to
generate a temporary string like "Foo=ARRAY(0xc8afd8)". I intend to merge
this after 5.26, and maybe booleanise a few more ops too (like length()).

The other notable thing I did last month was to change the output format
produced by the op_dump() function (and perl -Dx). It now displays an
ASCII-art tree structure, and more low-level information is displayed,
such as each op's address and class.

      3:21 RT #129285 Perl_gv_fullname4 assertion
      0:36 RT #130617 Perl_rpeep: Assertion ....
      0:55 RT #130621 Segfault in Perl_vwarner (util.c:2051)
      0:49 RT #130648 S_pat_upgrade_to_utf8 Assertion ...
      1:17 RT #130651] regcomp.c:6881 Assertion `expr' failed
      1:19 RT #130661 perly.c:341: Perl_yyparse: Assertion failed
      1:03 RT #130667 Perl_sv_clear Assertion `SvTYPE(sv) != ...
      0:52 RT #130669 Perl_sv_2nv_flags: Assertion `SvTYPE(sv) != ...
      5:18 [perl #130198] chop(@x =~ tr///)
      3:00 [perl #130584] Slow pos function in taint mode with \G
     22:53 [perl #78288] ref and other ops are inefficient in boolean context
     12:08 change output format of op_dump()
      3:20 factor out common optree finalisation code
      4:30 general housekeeping
     10:41 process p5p mailbox
      2:51 review security tickets
     74:53 TOTAL (HH::MM)

 172.3 weeks
2376.7 total hours
  13.8 average hours per week

There are 423 hours left on the grant

About TPF

The Perl Foundation - supporting the Perl community since 2000. Find out more at www.perlfoundation.org.

About this Archive

This page is an archive of entries from February 2017 listed from newest to oldest.

January 2017 is the previous archive.

March 2017 is the next archive.

Find recent content on the main index or look in the archives to find all content.


OpenID accepted here Learn more about OpenID
Powered by Movable Type 6.2.2