Maintaining Perl 5: Grant Report for November 2015

No Comments

Tony Cook writes:

Approximately 54 tickets were reviewed, and 12 patches were applied.

[perl #126593] illustrates how some of perl's internal tools need to be careful of which parts of the language they use.

The tr/// operator can do its job in one of two ways, if all the code points are between 0 and 255 with a 256 entry table of shorts, otherwise using a swash, which is created by SWASHNEW in lib/

tr/// uses the UTF-8 flag on the search and replacement strings to decide whether to use the look-up table or the swash, so it's possible for the swash to be used even when the search and replacement strings are representable as bytes.

aa8f6cef changed a s/// operator in lib/ to a tr/// operator. All of the characters in the search and replacement strings can be represented as bytes - they're all ASCII range, so at first sight the implementation should be using the lookup table rather than the swash.

The problem is code that uses the deprecated ${^ENCODING} variable, in this case the encoding::warnings module. encoding::warnings sets ${^ENCODING} to a filter that warns (or croaks) when a non-UTF-8 marked PV with non-ASCII is used with UTF-8 marked PVs.1

When parsing string literals, including tr/// operators, S_scan_str() in toke.c always returns UTF-8 marked strings when ${^ENCODING} is true

The module that started [perl #125693] loads encoding::warnings, so ${^ENCODING} is now set, then Fatal, which loads Carp which includes the line:

$VERSION =~ tr/_//d;

so both the search and replacement strings are passed to S_pmtrans() (op.c) as UTF-8 marked strings.

S_pmtrans() attempts to create a swash, which starts to load, until we get to the line:

(my $loose = $_[0]) =~ tr/-_ \t//d;

where things break.

[1] the current recommended practice is that the UTF-8 flag controls internal representation only and combining two such strings isn't an issue. Don't use encoding::warnings.

0.17#122251 review upstream tickets and resolve
0.45#123710 testing, close
2.55#123737 review discussion, testing
#123737 debugging, produce a patch and comment
1.62#123991 produce a patch and comment
2.51#124068 testing, fix issues, more testing
#124068 another fix, testing, push to blead
1.15#124080 review, testing
#124080 review test results, push to blead
2.32#124097 review code, try a fix
#124097 review more code, comment with patch
#124097 testing, comment
0.83#124349 produce a patch
0.75#125619 review discussion and comment
0.40#125830 review cflags.SH changes, testing, apply to blead
2.14#126042 research
#126042 more research, comment with a simple patch
2.86#126045 review discussion, research, work on test patch
#126045 more work on patch, comment with patch (and some
smartmatch irc discussion)
1.94#126193 review code, work on a patch, testing, comment with patch
#126193 re-check patch, testing, apply to blead
1.27#126240 (camel issues), testing and comment
0.52#126325 review patch, testing and apply to blead
0.50#126368 review, test all three modules with blead and close
0.53#126403 review and comment
0.23#126437 research, comment
2.32#126443 (sec) comment
#126443 create a test, testing, comment with new patch
#126443 testing, apply to blead
0.57#126469 review, re-test and apply to blead
1.37#126474 comment
#126474 research, comment
1.55#126480 produce a patch and comment
#126480 minor corrections to patch, push a smoke-me
#126480 testing, apply to blead
0.88#126502 testing
#126502 more testing, start bisect
#126502 comment
1.08#126533 review and comment
#126533 test and apply to blead
0.60#126534 review, test and apply
1.03#126544 research, comment
0.40#126546 fuzzer tickets meta-ticket
0.60#126552 testing, debugging
5.59#126593 debugging, research, longer comment
#126593 review discussion, try some fixes
#126593 comment with patch
2.03#126602 review, produce a patch and comment
#126602 re-check patch, testing, apply to blead
0.50#126608 review and comment
0.43#126609 review, test and apply to blead
0.76#126611 review and comment
#126611 review latest patch, testing, apply to blead
3.13#126621 review, code, prep for testing, reproduce, review
failing tests, work on simple reproducer
#126621 comment, try to work out a fix
0.15#126632 review, requires CPAN updates, so leave
2.17#126633 review failing code, review patch, make a simple reproducer
#126633 try to make a simpler reproducer, try a simple
patch and comment
1.93#126635 reproduce, review code, produce a patch and comment
#126635 produce alternate patch and comment
#126635 testing, apply to blead
0.98#126707 review change, research, comment silent handling, notice unneeded rebuilds and
open #126710
1.67#126719 debugging, open Encode ticket, comments
0.32#126731 review code, testing, find dup and comment, merge tickets
3.64#126755 (sec) try to map MapPathW crash, working up test code
#126755 (sec) more test code, testing, re-working supplied fix
0.355.24.0 blocker updates
7.01#57512 review discussion, look over code
#57512 more look over code, testing
#57512 research, code on in-place edit close failures
#57512 more code, testing, comment with patches
2.78cygwin build warnings - testing, seem to be 32-bit
generic, testing, fixes
0.32cygwin failures – re-test original fix and apply to blead
0.48jhi's scandir thread – research and comment
0.63khw's cygwin issues
0.28os390 dynaloader issue: fix an else nesting bug
0.65rjbs's redefinition warning strangeness: irc and reply to list
0.93smartmatch review overloading code
2.47smartmatch testing, work on overload.t failures
2.08smartmatch: fix overload.t tests, more testing, push to branch
2.72smartmatch: rebase, work on overloading

78.39 Hours Total

Leave a comment

About TPF

The Perl Foundation - supporting the Perl community since 2000. Find out more at

About this Entry

This page contains a single entry by Karen Pauley published on January 27, 2016 4:57 AM.

Migrating Second Progress Report was the previous entry in this blog.

Grant Report: Test::Simple/Stream Stabilization is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.


OpenID accepted here Learn more about OpenID
Powered by Movable Type 6.2.2