Maintaining Perl 5: Grant Report for November 2015

Tony Cook writes:

Approximately 54 tickets were reviewed, and 12 patches were applied.

[perl #126593] illustrates how some of perl's internal tools need to be careful of which parts of the language they use.

The tr/// operator can do its job in one of two ways, if all the code points are between 0 and 255 with a 256 entry table of shorts, otherwise using a swash, which is created by SWASHNEW in lib/utf8_heavy.pl.

tr/// uses the UTF-8 flag on the search and replacement strings to decide whether to use the look-up table or the swash, so it's possible for the swash to be used even when the search and replacement strings are representable as bytes.

aa8f6cef changed a s/// operator in lib/utf8_heavy.pl to a tr/// operator. All of the characters in the search and replacement strings can be represented as bytes - they're all ASCII range, so at first sight the implementation should be using the lookup table rather than the swash.

The problem is code that uses the deprecated ${^ENCODING} variable, in this case the encoding::warnings module. encoding::warnings sets ${^ENCODING} to a filter that warns (or croaks) when a non-UTF-8 marked PV with non-ASCII is used with UTF-8 marked PVs.¹

When parsing string literals, including tr/// operators, S_scan_str() in toke.c always returns UTF-8 marked strings when ${^ENCODING} is true

The module that started [perl #125693] loads encoding::warnings, so ${^ENCODING} is now set, then Fatal, which loads Carp which includes the line:

$VERSION =~ tr/_//d;

so both the search and replacement strings are passed to S_pmtrans() (op.c) as UTF-8 marked strings.

S_pmtrans() attempts to create a swash, which starts to load utf8_heavy.pl, until we get to the line:

(my $loose = $_[0]) =~ tr/-_ \t//d;

where things break.

[1] the current recommended practice is that the UTF-8 flag controls internal representation only and combining two such strings isn't an issue. Don't use encoding::warnings.