Recently in Perl 5 Development Category

This is a monthly report by Dave Mitchell on his grant under Perl 5 Core Maintenance Fund. We thank the TPF sponsors to make this grant possible.

The main things I did last month were:

Firstly, fixing various issues with scopes in regexes. In particular,
(RT #126697), code blocks sometimes failed to undo localisations when
backtracking. For example the $s below wasn't always being restored when
the B part of the match failed and it backtracked to try another A - where
A represents something complex like (\w+\s+)* which can match in multiple
ways:

    /A(?{ local $s = ...})B/

As part of that work, non-greedy matching of complex sub-expressions with
captures and repeated backtracking was made more efficient under some
circumstances; for example the following now runs about 25% faster:

    $s = ("a" x 1000);
    $s =~ /^(?:(.)(.))*?[XY]/ for 1..10_000;

Secondly, improving 't/TEST -deparse'.

The -deparse option to t/TEST causes it to run all the core's test
scripts, but after running them through the deparser first. Many of these
modified scripts are currently known to fail, and there is an exclusion
file, Porting/deparse-skips.txt, which is supposed to list the known
failures. However, over time, new failures have appeared which are not in
the exclusion list. Last August I did some work on Deparse.pm and managed
to reduce some of the expected and unexpected failures, but since then
more failures have crept in.

My recent work includes: modifying t/TEST so that it distinguishes
between expected failures and unexpected passes, and warning of unknown
files in Porting/deparse-skips.txt; purging Porting/deparse-skips.txt to
account for files that have been renamed or are no longer in the core, and
to reflect the current state of things; and fixing Deparse.pm to:
    * better handle 'BEGIN { use_ok() }';
    * better handle 'BEGIN { require expr }' (as opposed to require Bareword);
    * deparse lexical var attributes, e.g. 'my $foo :bar(baz)';
    * avoid a 'deep recursion' warning;
    * handle an escaped literal tab char in a pattern, e.g
        /.....\ ..../x where the whitespace char following the backslash is
        actually a tab; previously the deparse failed to emit the backslash;
    * handle declarations with multiple lexical vars within a pattern code
      block, e.g. /(?{ my ($x, $y) = @a; })/

Because we're currently in code freeze, this as been pushed as
smoke-me/davem/deparse and will be merged in 5.27.x.

Thirdly, reviewing and fixing tickets in the security queue. There's quite
a lot of tickets in the security queue due to fuzzing, where if the fuzzer
detects a use-after-free or buffer overrun for example, the reporter
submits it to the security queue rather than the normal queue. Once
examined, 95% of the time it will be found to be harmless or
non-exploitable, but until someone has assessed and fixed it, it lingers
as an open security ticket.

Perl 5 Core Hackathon (p5hack) Report

No Comments

perl5-hack-logo-04.jpg

(Graphics credit to Leonardo Maia.)

Thanks to The Perl Foundation and our beloved sponsors, a large portion of the critical contributors to the Perl 5 Porters (p5p), the core group of Perl 5 developers, were able to get together on November 11th, 2016 at the Booking.com headquarters in Amsterdam, The Netherlands, for our very first Perl 5 Core hackathon, nicknamed Perl 5 Hack.

This is our report.


It was a four-day event, composed of discussions (centralized and specialized) on a myriad of important topics, knowledge sharing, and cooperative hacking. It provided us with the ability to cut down several months worth of discussions and loss of time due to timezone differences.

We worked on several topics, such as the Configure phase, core modules, security updates to Perl library loading order ("dot in @INC€), improving compilation under C++11, Core API and internals, replacements for the buggy utf8 IO layer, conditional lexical variable warnings, and deparsing some internal warnings information (${^WARNING_BITS}).

This alone would have been quite the fit, but we also achieved a few things along with those discussions. Some of our achievements include:

  • Cleaning up dead hints code (pre-ELF)
  • Improved our release guide
  • Improved our Configure process
    • Removing hundreds of complicated C conditionals
    • Faster Configure script
  • Cleaned up our install output
  • Improved our API documentation
  • Massive speed improvements to core modules
  • Explored the new Test2 testing framework for Perl
  • Reviewed and listed all current deprecations
  • Tied all deprecations to an exact version, to improve transparency and communicate better our intentions regarding deprecated features and syntax
  • Cleaned up core code:
    • Removed obscure and unused SVs
    • Removed dead code
    • Moved Unicode::Normalize to upstream blead
  • Had done optimizations, such as when assigning to references.
  • Reduced memory usage in core modules: Data::Dumper and Time::Local.
  • Fuzzed core modules
  • Have rewritten much of our readline implementation
  • Introduced the indented HEREDOC feature, being released in the upcoming 5.26

We raised a lot of topics and we made numerous important and valuable changes to the core code and core utilities, but we have also held in-depth discussions on the following topics:

  • Our COW implementation and where to take it in the future
  • Efficiency of our hash bucket sizing
  • Overloading indication
  • Our policy on editing dual-life modules
  • OP tree cloning
  • Subroutine Signatures and their introspection
  • The internals of the Perl to C compiler and what may be gleaned from it on improving perl
  • The contribution hurdles when working with RT
  • Vulnerability triage, classification, and handling - covering our security handling policies
  • EBCDIC-portability
  • Unicode (and specifically grapheme delimiters)
  • Regular Expression internals
  • Vtable-based hashes

Most of these have led to changes in policy, improvements or cleanups in core, and a "leg-up" on additional research into those topics.

All in all, despite it only being the first hackathon, it had already proved successful and beneficial. For this reason we intend to holding future hackathons, while both preparing better for them, as well as communicating better our work on them onward to the community.

Thank you to all attendees: Aaron Crane, Abigail, Brian Fraser, Dagfinn Mannsåker, Dave Mitchell, Dennis Kaarsemaker, Gonzalo Diethelm, H. Merijn Brand, John Lightsey, J. Nick Koston. Karl Williamson, Leon Timmermans, Lukas Mai, Matthew Horsfall, Mattia Barbon, Nicholas Clark, Nicolas Rochelemagne, Sawyer X, Steffen Mueller, Stevan Little, Todd Rinaldo, Viekntiy Fusenov, and Yves Orton,

And thank you to The Perl Foundation for helping in organizing this event, and to all of our sponsors for contributing and assisting in this event: Our Diamond Sponsors, Booking.com; our Platinum Sponsors of cPanel and Craigslist; Silver Sponsors, Bluehost; and Bronze Sponsors of Assurant, Grant Street Group and Signature Information Solutions, and to FastMail, for their support of this, and future Perl Foundation activities.

A special thank you to the Perl community for the support it has given us and to the Perl 5 Porters group.

Sawyer X and Todd Rinaldo, Perl 5 Hackathon organizers.

Dave Mitchell writes:

I spent December:

1) looking for quick some wins on speeding up perl compile-time, using
'perl -MCPAN -e1' as a typical example of loading and compiling several
.pm files.  I tweaked Perl_yyparse() and shaved ~2% off the compile time;
then tweaked Perl_sv_gets() and shaved another ~2% off, and also got a
pleasing run-time boost for line reading, with reading a big list of words
now ~8% faster on my system:

    perl -e'$i++ while (<>)' /usr/share/dict/words

2) investigating possible ways of making the regex engine faster by (in
some cases) processing 8 bytes at a time on 64-bit platforms. This
culminated in an this email thread:

    http://nntp.perl.org/group/perl.perl5.porters/241891

3) as usual, working on a few miscellaneous tickets.

SUMMARY:
      1:30 RT #130385 Bleadperl breaks DNS-LDNS
      1:00 RT #130385 Bleadperl breaks List-Pairwise
      1:00 RT #130398 Bleadperl breaks Method-Signatures
      4:26 [perl #129199] Difficult-to-trigger panic
      1:01 [perl #130247] Perl_rpeep(OP *): Assertion `oldop' failed
      0:55 [perl #130307] Bug: Regex matches when it shouldn't
      0:33 [perl #130311] heap-buffer-overflow Perl_yyparse
      0:55 fix build warnings and smoke failures
      1:40 fix some CPAN modules under PERL_OP_PARENT
     19:00 investigate making find_by_class faster()
      2:37 optimise Perl_sv_gets()
      4:45 optimise Perl_yyparse()
     12:20 process p5p mailbox
      3:22 review RT tickets
      3:04 review security tickets
    ------
     58:08 TOTAL (HH::MM)

 170.1 weeks
2338.8 total hours
  13.7 average hours per week

There are 61 hours left on the grant

Dave Mitchell writes:

I spent October mainly working on two things.

First, I optimised some common arithmetic operators: + - *, so that for the very common case of both args being simple ints in ranges that won't overflow, or both being floats, a simple C-level + or whatever can be directly done. For more complex or mixed args, it falls back to the existing slower code. For ++ and --, I optimised the simple integer case. I also improved the core SET[iun], PUSH[iun] and XPUSH[iun] macros, which set the pad targ to a numeric value and push it on the stack. Since PADTMPs and lex vars (which is typically what a pad targ is) are often used in the same way, e.g. always assigned and used as an integer, those macros now check whether the targ is already of the right type, and if so directly set the value, rather than just blindly calling sv_setiv() etc. The combination of the above makes the nbody benchmark (lots of floating-point vector arithmetic) about a third faster.

Second, I continued overhauling perl's context stack and dynamic scope implementation.

Summary

59:16 #124156: death during unwinding causes crash
0:09 [perl #117341] av_undef's POD is confusing
0:33 [perl #125937] 'x' operator on list causes segfault with possible stack corruption
4:35 [perl #126082] unshift to @ISA
1:11 [perl #126145] Problem with stack moving fix for Perl_load_module
2:23 [perl #126170] Assertion failed: S_finalize_op (op.c:2562)
1:55 [perl #126229] POSIX::strerror() clears $!
2:27 [perl #126309] 'x' operator on list causes segfault and confuses valgrind, 64-bit version
0:29 [perl #126472] Bleadperl v5.23.3-33-g6768377 breaks HANENKAMP/Tie-Simple-1.03.tar.gz
0:32 add perldelta entries
13:00 make arithmetic faster
0:34 optimise the Boyer-Moore string finder (as used in REs and index())
11:45 process p5p mailbox

98:49 Total (HH::MM)

As of 2015/10/28: since the beginning of the grant:

106.4 weeks
1600.0 total hours
15.0 average hours per week

There are 0 hours left on the grant.

Dave Mitchell writes:

I spent July mainly working on two things.

First, I continued to work on the whole issue of how subroutines are invoked and returned from, and especially how the various perl stacks are manipulated during this time, i.e. all the PUSHBLOCK/PUSHSUB stuff. I also started extending the work to other context types, such as loops.

I have mainly concentrated on removing unnecessary fields from the CXt_SUB context struct, and eliminating the ENTER / SAVE* .... / LEAVE that is normally wrapped around function calls, loops etc, instead storing the old save stack and tmps stack floors and old PL_comppad directly in new fields in the context struct. I've also streamlined the @_ processing and tweaked pp_entersub a lot.

I'm delighted to announce that Dave Mitchell's recent grant extension request has been successful. This extension will allow Dave to continue with this work for another 400 hours.

Thanks to everyone who provided feedback on this application and to all those who continue to provide financial support to the Perl 5 Core Maintenance Fund. If you are interested in funding this work please contact karen [at] perlfoundation.org or donate directly through our donation system.

Booking.com Donates $60,000 to Perl 5 Development

No Comments

It is our pleasure to announce that Booking.com has donated the sum of $60,000 to the Perl Foundation to aid with further development of the Perl 5 programming language. This donation is a further contribution to the Core Maintenance Fund (P5CMF) that Booking.com has consistently supported over several years. This donation is a another step in the support that Booking.com gives to TPF and the broader Perl community.

Darren Huston, Chief Executive Officer of Booking.com, said "Perl is central to the success of Booking.com." By continuing with grants, supporting Perl community events and hackathons:

"we keep up our tradition of giving back to the community. As one of the world's largest employers of Perl developers, an investment for us into a healthy and dynamic Perl community is an investment in the future of our company and innovative web development in general."

Booking.com has also recently entered into an arrangement to continuously fund some of David Mitchell's work on the Perl 5 Core. This has a specific focus towards performance improvements and evolution of the language in line with the vision of the project leader. This future-focus secures Perl's relevance as an enterprise grade language for large web applications.

"It is always a tremendous pleasure to work with Booking.com," said Karen Pauley, President of the Perl Foundation.

"With Booking.com it goes far deeper than a desire to promote our language or support our community, it is a genuine feeling that they wish to belong which is why they involve themselves so much. They are a part of the Perl family."

Since the foundation of Booking.com, in 1996, the company has built its IT infrastructure on myriad of open source projects and products. These include Linux, MySQL, Nginx, Hadoop, and Redis to complement a strong association with the Perl programming language. Because of this, Booking.com is a keen supporter of the Perl and open source communities. As Steffen Mueller, Senior Manager for Technology and Infrastructure at Booking.com, would say:

"Developers at Booking.com are encouraged to release code to Open Source projects. Booking.com has a culture of contributing to the Open Source community. Our staff are regular attendees and speakers at Perl events, but also find themselves at non-Perl conferences. We are happy to hire talented developers from a variety of backgrounds and varying skill sets and re-train them in the Booking.com culture and Perl language."

booking_logo.png

About Booking.com
Booking.com B.V., part of the Priceline Group (Nasdaq: PCLN), owns and operates Booking.comâ„¢, the world leader in booking accommodation online. Each day, over 700,000 room nights are reserved on Booking.com. The Booking.com website and apps attract visitors from both the leisure and business sectors worldwide.

Established in 1996, Booking.com B.V. guarantees the best prices for every type of property, from small, family-run bed and breakfasts to executive apartments and five-star luxury suites. Self-catered properties are also specifically a focus of Villas.com, a sister site recently launched by Booking.com. Truly international, Booking.com is available in more than 40 languages, and offers over 537,980 properties in 207 countries.

About TPF

The Perl Foundation - supporting the Perl community since 2000. Find out more at www.perlfoundation.org.

About this Archive

This page is an archive of recent entries in the Perl 5 Development category.

Parrot development is the previous category.

Perl 6 Development is the next category.

Find recent content on the main index or look in the archives to find all content.

Pages

OpenID accepted here Learn more about OpenID
Powered by Movable Type 6.2.2