July 2012 Archives

Patrick Michaud writes:

This is an interim report for my Hague Grant on Lists, Iterators, and Parcels. This report has been far too long in coming; at the time of the original grant proposal (July 2010) it was expected that the design and implementation of Perl 6 lists and iterators was near completion, and that most of what remained to be done would be updating the documentation, tests, and Synopses.

Since that time we've essentially learned that the design as of July 2010 still had some holes in it, especially in the area of performance, and thus I have been working to solidify the underpinnings of both. The lists and iteration implementation in Rakudo 2012.07 is some 40 to 50 times faster than the one that existed at the time of the original grant proposal, and is now written almost entirely in Perl 6 instead of PIR.

As of this report, we now have an updated draft of Synopsis 7 that reflects the best understanding of Perl 6 lists and iterators as of July 2012. In addition, I have given several presentations at YAPC::EU 2011 and YAPC::NA 2012 (1 2) on Perl 6 lists and iteration. The draft is still a draft, even as recently as June 2012 there have been some questions regarding the funamental semantics of Perl 6 lists that may affect the ultimate design. I am currently working with people on #perl6 and the Perl 6 design team to resolve the remaining issues and have them fully documented.

Here is the status of the deliverables identified in the original report:

D1: (Completion of the new list implementation in Rakudo Perl 6.) The list implementation referred to in July 2010 was essentially completed in the following month, but it became quickly apparent that it had performance problems. In Spring 2011 the list code was completely reimplemented in Perl 6 as part of the "nom" branch of development and was about 2x faster than previously. Since then several improvements and refactors have been made to basic list and map operations such that list handling is now 40x to 50x faster than it was in 2010.

More changes are left to be made; there are still some spec-related questions regarding the handling of Parcel flattening and combining of lists, which are waiting on input from the Perl 6 design team. I expect these to be resolved within the next six weeks.

D2: (Review and updated tests for iterables and lists in official test suite.) All of the existing tests have been updated for compliance with the current Lists and Iterables implementation and description. A new section in the test suite is being created to specifically test the features defined by the new Synopsis 7 draft. A key focus of this additional section will be to explicitly document and enforce the list-related design decisions that have been made by the design team over the past several years of development.

D3: (A new version of Synopsis 7.) A draft of a new Synopsis 7 has been checked into the perl6/specs repository. [3] Over the next couple of weeks it will be enhanced with detailed descriptions of List-related methods and functions, as well as incorporate any new decisions made by the design team.

D4: (Update Synopsis 8.) After working on drafts of Synopsis 7, it seems that it will make more sense to document the Parcel and Capture classes directly in Synopsis 7, rather than in a separate synopsis. The handling of slices has changed somewhat since July 2010, and now belongs more naturally in Synopsis 9 (which already covers a fair bit on slicing). The end result of this grant will likely be to eliminate the current Synopsis 8 document, leaving the slot open for documentation of some other Perl 6 feature dealing with data structures (perhaps hash-related items will go here).

D5: Updates to Synopsis 32 will be held until D1, D2, and D3 above are nearing completion.

D6: As mentioned earlier, several presentations have been made regarding List implementation in Perl 6; the material from these presentations will be translated into blog posts over the next couple of months (and updated to reflect changes from deliverables above).

Rakudo users have all remarked at the improvements in list handling over the past couple of years; Rakudo lists now handle infinite sequences and lazy behaviors in a much more intuitive manner than existed in July 2010. What remains to do in this grant is to iron out a few of the remaining details with the design team, document them, add more tests reflecting the decisions made, and write up the final reports and blog postings. In spite of the delays that have occurred thus far with this grant, I believe we are now at a point where the remainder of this grant can be completed within the next two or three months.

Josh McAdams is stepping down as conferences chair, so TPF is now accepting self-nominations for this role. This is a great opportunity to help the Perl community and is one of the key roles that can have a huge impact without writing code. You can take a look at the expectations for the role on the conference roles page. If you're interested, answer the questions at the bottom of the page and send it to me at cbrandt at perlfoundation dot org.

TPF committees typically select their chairs, so I'll forward all submissions on to the conferences committee for discussion and selection. The deadline for submissions is Wednesday, August 15, so get started!

Thanks to Josh for all of his work!

Joel Berger wrote:

After another busy month outside of the Perl world, I have gotten a little more time in the last week to work on Alien::Base. I must especially thank fellow WindyCity.pm member David Mertens for working with me on some of the Mac problems involved.

N.B. I also want to thank him for PDL::Graphics::Prima, which made the rest of my $work easier this month! If you need interactive plotting, give it a look! Also Chicagoans, WindyCity.pm is tomorrow, topic: Dancer.

Ok so after much tinkering, I have made enough changes to have the test suite passing on Mac, however, it gives warnings which make me think that while it is "working" in the blib directory, that it won't work once installed. To this end I have been putting together some "real" Alien::Base modules to be released into the ACME:: namespace (if you care, look for ACME::Alien::DontPanic (which consumes Alien::Base) and ACME::Ford::Prefect which depends on it). To make this a more real test case, I have been working to convert my libdontpanic shared library to the proper autotools chain. I have spent the day learning about autoconf/automake/libtool etc, and let me say, if you never need to learn this, then consider yourself lucky!

Hopefully these test modules will illuminate the real problems in relocating the .dylib files, and perhaps other non-Mac problems too!

Original article at Joel Berger [blogs.perl.org].

Nicholas Clark writes:

The most significant activity for the month was discovering, investigating and iterating fixes for the various bugs now fixed on the branch smoke-me/require. This is the cause of the delay on making a 5.16.1 release, as Ricardo explained in http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2012-07/msg00954.html and hence this report, as the issue only went public two days ago.

There are two partly related problems. Both affect code which takes untrusted user data and passes it to routines that load code (a risky thing to do at the best of times), but doesn't perform sufficiently strong validation. The called routines could detect that the passed in string is crazy, but don't, and instead proceed to treat it in surprising ways.

Note, this only affects programs that

1) are "brave" enough to load code from disk using package names provided by user data
2) don't validate that user data thoroughly

(ie a generally bad idea, and a definitely bad implementation)

The first is accessible from Perl-space:

$ cat /tmp/foo.pm
print STDERR __FILE__ . " was not the file you expected to get loaded?\n";
$ ~/Sandpit/5000/bin/perl -e 'require ::tmp::foo;'
/tmp/foo.pm was not the file you expected to get loaded?
$ echo $?

and is present for every version of Perl 5 ever. (Perl 4 is safe)

Of course, to be bitten by that you have to pass user data to a string eval containing require, which is already a risky thing to be doing, and this only increases the attack space from "all readable files in @INC ending .pm" to "all readable files on disk ending .pm"

Plus if you're already passing user data to string eval, it's usually much easier to do direct nasties - eg strings such as "less; system 'sudo rm -rf /'"

So this probably doesn't expose any holes in programs that weren't already completely unseaworthy.

It's also rather hard to search CPAN to work out if there's any code which is unwise enough to be at risk from this.

The second is accessible only to C and XS code that call the APIs function Perl_load_module() or Perl_vload_module(). These are documented as taking a module name (ie "Foo::Bar"), not a file name (ie not "Foo/Bar.pm"). However, they don't validate what they are passed, so module names such as "Foo::..::..::..::Bar" are subject to the standard s!::!/!g transformation, which obviously can be used to produce "interesting"

Perl_load_module() has been available since 5.6.0.

Note, again this only affects programs that

1) are "brave" enough to load code from disk using package names provided by user data
2) don't validate that user data thoroughly

(ie a generally bad idea, and a definitely bad implementation)

and again this only increases the attack space from all readable files in @INC to all readable files on disk.

Nothing on CPAN uses Perl_vload_module(), and (as best I can tell from searching) only two modules on CPAN use Perl_load_module() on values that aren't constants or tightly controlled at the C level. The authors of both modules are aware, and have checked their code.

While investigating the problem described last month, of testing as root but the file tree owned by a different user, I discovered a different problem with File::stat's tests when running as root. This had prompted me to look at its regression test, to get a feel for why this wasn't spotted before. At which point the scale of the job escalated, because the test didn't look that robust or as complete as it could be. So at the start of the month I started on refactoring it, to give increase my confidence in it actually being able to spot bugs. In the process I discovered that just like t/op/filetest.t it had accumulated quite a lot of layers of cruft, each attempting to solve one of the various problems that had emerged over the years, but none looking closely at what the intent other work arounds were, and whether combination could be simplified.

In this case, again the problems related to "which is our victim file?" Initially the test used t/TEST. But (again), that file might be a symlink (if built with -Dmksymlinks) so the setup code has to check for that, and if so, chase the symlink back to the original file. All good so far. But, it gets worse: the test checks the output of stat, which includes the last access time, and with parallel testing, other files can read that. So the script was changed to use itself as the victim. However, the original code made the "golden result" stat call in a BEGIN block, which can still go wrong because BEGIN blocks run during compilation, and it may be that the interpreter hasn't finished reading the entire file yet, in which case the atime will get updated.

The simple solution is to use a tempfile.

This also has the benefit of being able to change the file's mode, and hence test many more permutations of the various file operators. This seemed like a quick job, so I pushed another iteration of File::stat test improvements a smoke-me branch, so that George Greer's Win32 smoker could take it for a spin.

Sigh. These things always take longer than you think. So, the aim was get the code to check filetest operators with various different file modes, not just the mode the tempfile happens to be created with. But, while refactoring the code to make this work, I realised that some of the other tests weren't actually doing what they thought that they did. (Inconceivable!)

Specifically, the end of the test script contains code intended to test that bug ID 20011110.104 (RT #7896) is fixed. The bug was indeed fixed by the changes to File::stat made by commit 2f173a711df6278f in Nov 2001, but the test that commit added never actually tested this.

The initial problem was that the new code, as written, used stat, intending that to call File::stat::stat(). However the refactoring of the test script (all part of the same commit) from use File::stat; to use_ok( 'File::stat' ); (not in a BEGIN block) intentionally eliminated the export of &File::stat::stat. This meant that the plain stat the added code used was the core builtin. The core builtin never exhibited the bug. :-)

Fixing this as-is to File::stat::stat() won't help, as tests have subsequently been added earlier in the script that trigger the autoloading of Symbol by File::stat (commit 83716b1ec25b41f2 in Feb 2002). Moving the tests earlier won't help now that the test uses File::Temp, as that loads IO::Seekable, which loads IO::Handle, and that unconditionally loads Symbol.

So the simplest solution seemed to be to move the test to its own file, which I did. Now, the bug is actually tested for, and regressions should not be possible. (But make it foolproof, and they build a better fool.)

So, finally, I got to the point where I could fix a long-standing bug in File::stat's handling of -x and -X for directories and executable files when running as root, which I'd spotted last month.

Previously File::stat's overloaded -x and -X operators gave the correct results only for regular files when running as root. However, they had been treating executable permissions for root just like for any other user, performing group membership tests etc. for files not owned by root. They now follow the correct Unix behaviour - for a directory they are always true, and for a file if any of the three execute permission bits are set then they report that root can execute the file. Perl's builtin -x and -X operators, added in Perl 2, have always been correct.

Thinking I was home and dry, I pushed a smoke-me branch to the repository and waited for the testers on various platforms to check it out and report back. Result - nearly everything worked, except for a mysterious failure on OS X. So I tried the test on OS X locally, and obviously I couldn't replicate the failure. Which resulted in a careful scouring of the smoker logs (exactly which test did fail?), trying to replicate the problem on various other operating systems (no success), and eventually realising that the problem was a combination of two things. Firstly that darwin defaults mounts to POSIXly-correct atime semantics - atime is always updated when a file is read. Secondly, when tests run in parallel, the atime of $^X will update whenever another perl process starts. The result is a race condition between the stat calls in this test, and any other test that runs. Life is always fun when reads trigger write actions - and it doesn't just happen with perl's scalar types.

It also turned out that my refactoring of the test exposed a latent bug present in it, which I failed to spot. Fortunately Father Chrysostomos diagnosed that the test was always using stat(), even when testing -l, and fixed it to use lstat() in that case.

I spotted some code in Class::Struct related to 5.002 and 5.003 compatibility. Obviously that's long dead, so it should come out, and this will be a nice quick one-hit tidy up. Of course, the world doesn't work like this. The code in question affects an error message, and it turns out that there are no tests for any of the error conditions in Class::Struct. It felt wrong to add just one test for the error check whose code I was refactoring, leaving all the rest naked, so the simple fix acquired a not-so-simple pre-requisite. And then that turned out to reveal another hole in the testing - initialiser lists had insufficient tests. So, before testing the error paths in initialiser lists, we probably should test the success paths. And so some time later, I removed (net) 5 lines of dead code. Collateral damage - 69 regression tests, where previously there were only 26.

Config::Perl::V is a module written by H. Merijn Brand to provide structured access to the output of perl -V, used by most of the CPAN testing infrastructure. The reason that it exists is that historically some configuration information about the running perl had only been available in the output from running perl -V. This gets "interesting" if the running perl program wants the values, as not only does one have to parse the output robustly and cope with the obscure case, but in the general case re-running the correct perl interpreter binary is not a trivial task - possibly it's not installed so library paths need overriding, or possibly not the first perl in the PATH. The module abstracts all this away nicely.

The plan is to add it to the core distribution, because it simplifies the CPAN toolchain to have it always available. However, the "historically" above is pertinent - the implementation of Config::Perl::V predates changes I made for 5.14.0, which has functions in the Config module which expose to Perl space all the information previously only available by running perl -V. Hence I spent some time updating Config::Perl::V to use the routines in Config where available, and adding a test that cross checks that the old and new approaches give the same answers.

Blead had been failing two tests when built "outside" the source tree using a symlink forest generated by -Dmksymlinks. In both cases, they related to how various porting tests that require a git repository work under -Dmksymlinks. To enable the tests to run instead of simply skipping, the setup routine would detect the symlink situation, and that they pointed to a real checkout with a .git directory, and change directory to it, so that git commands would work.

However, running the tests outside of the build tree turns out also to cause problems. t/porting/cmp_version.t was failing because it could see that ext/DynaLoader/dl_vms.xs has been modified since the last tag, but was unable to work out which was the related .pm file, to validate that $VERSION had been increased. It turned out that this was working just find inside a build tree, because ext/DynaLoader/DynaLoader.pm exists there, having been created by ExtUtils::MakeMaker from ext/DynaLoader/DynaLoader_pm.PL However, in a clean tree, it doesn't exist, hence the confusion. So I improved Porting/cmpVERSION.pl (the underlying driver script) to teach it about _pm.PL files.

t/porting/utils.t had recently started failing, because Ricardo changed it to avoid false positive failures, by only running if it found itself in a git checkout. However, it assumes that it can find many files generated by the build process (to syntax check them), whilst the "are we in a git checkout?" code assumes that it should change directory to the checkout, if one is building symlinked from it. Result - various files that "should" be there are now missing. The solution was to change the "are we in a git checkout?" code to set GIT_DIR instead of changing directory, and return the path to the git checkout. That way, tests that really need to be in the git checkout (ie t/porting/cmp_version.t) can change directory to it, but by default tests continue to run from the build tree. This has two benefits

a) fewer surprises
b) for a couple of tests, we can drop the code that converts @INC to absolute
paths, because they no longer change directory

(The tests by default run, and have always run, with the relative path of the build lib/ in @INC. Converting it to an absolute path relies on fairly complex platform specific Perl code, and how are you going to safely test that works before you're even sure that simple Perl code works? When testing an interpreter from scratch, you need to bootstrap your tests too, because you initially can't assume anything works.)

A more detailed breakdown summarised from the weekly reports. In these:

16 hex digits refer to commits in http://perl5.git.perl.org/perl.git
RT #... is a bug in https://rt.perl.org/rt3/
CPAN #... is a bug in https://rt.cpan.org/Public/
BBC is "bleadperl breaks CPAN" - Andreas König's test reports for CPAN modules
ID YYYYMMDD.### is an bug number in the old bug system. The RT # is given
afterwards. You can look up the old IDs at https://rt.perl.org/perlbug/

0.75HP-UX -Duse64bitall
0.50ID 20020306.011 (RT #8800)
0.50Module::Build failures on VMS
3.00Porting tests with -Dmksymlinks
0.50RT #113094
0.25RT #113464
0.75RT #113472
0.25RT #113620
0.25RT #16249
0.25Ubuntu link fail with non-English locales
0.25applying patches (with massaging)
0.75bisect.pl (Debian multiarch)
0.25mktables memory usage
0.75named arguments in prototypes
0.25perldelta updates
1.00process, scalability, mentoring
1.75re/eval on VMS
23.25reading/responding to list mail
0.25regexp set syntax
1.00smoke-me branches
2.00smoker logs
t/op/arith.t tryeq_sloppy

102.50 hours total

YAPC::NA 2013 Will be in Austin

I'm happy to announce that YAPC::NA in 2013 will be in Austin. We had two great locations to choose from, especially from the perspective of other interesting things to do outside the conference. Details and dates will be coming shortly. Congratulations to the Austin group!

Austin YAPC::NA 2013 Bid

Posting the bid submitted by Erica Baum for Austin to host YAPC::NA::2013.

Austin YAPC Proposal.pdf

Orlando YAPC::NA 2013 Bid

Posting the bid submitted by Mark Prather for Orlando to host YAPC::NA::2013.

yapc2013 proposal.pdf

Devel::Cover Grant Report for June

by Paul Johnson

In accordance with the terms of my grant from TPF this is the monthly report for my work on improving Devel::Cover covering June 2012.

This month I released Devel::Cover 0.88 and 0.89.

I looked into https://rt.perl.org/rt3//Public/Bug/Display.html?id=113464 which is a perl bug, but is related to warnings when running Devel::Cover. I supplied a core patch to quieten the warnings which has been applied, but I also changed Devel::Cover to turn off warnings when calling code in B::Deparse since the consensus seems to be that we probably have better things to do with our time than ensure that B::Deparse is warnings free.

I spent some time on cpancover. I received some feedback from my initial work with suggestions for improvements, which I implemented. I also automated the process of generating the coverage run with a view to running it once or twice a week from cron. Perhaps it could even run daily now that I have also compressed the results and configured nginx to serve those.

At the moment the most recent results are found at
http://cpancover.com/latest/index.html which is redirected to from http://cpancover.com but I need to come up with a main page linking to the available versions and perhaps some way of keeping links alive for a reasonable amount of time.

It would also be nice to get something like Thomas Klausner's App::ArchiveDevelCover (https://github.com/domm/App-ArchiveDevelCover) running to track coverage changes to modules. I have got many more ideas, but they are for another day.

I also tested Devel::Cover against 5.17.0. I had some cpantesters failure reports against 5.17.0 but I was not able to reproduce them, so I will need to look further into that.

I looked into perl #113690 and #113684 which were about perl commit 6a31dbf44ee919c340a3372c95b28d581979d165 breaking Devel::Cover 0.89. This commit removed some superfluous parentheses from B::Deparse output which caused Devel::Cover output to change and hence tests to fail.

I had planned to fix this as soon as perl 5.17.1 was released, but Father Chrysostomos realised that the fix was incomplete and needed further work to get the precedence levels correct. So he reverted the change and suggested I wait until the complete fix is committed.

As to why I was planning on waiting until 5.17.1 was released before fixing Devel::Cover, perl is a moving target as far as Devel::Cover is concerned and I choose to only worry about stable releases since 5.6.1 and development releases with version numbers above the most recent stable release. Arguably, that's already too many versions, but it's not causing a problem at the moment.

As expected 5.17.1 was released, and Devel::Cover passes all its tests with it.

And since I've started working on this grant a few people have approached me with ideas and suggestions, and with offers of help. This is great and something I had hoped would happen. So I have also spent a little time discussing these things and making future plans.

The work I have completed in the time covered by this report is:

Closed RT tickets:

  • 75633 Directory names with ++ not supported
  • 77598 [PATCH] Fix test failures when the build directory contains regexp metacharacters.
  • 77599 [PATCH] POD errors
  • 61515 Cover.pm messes with cwd
  • 70162 cover fails with spaces in pwd
  • 69562 cpancover won't run without Parallel::Iterator

Closed Github tickets:

  • 16 cover -coverage documentation
  • 17 #line directives
  • 19 Add some math modules
  • 21 Add links from subroutine/branch etc reports to the main report file
  • 22 Database permissions

Github tickets worked on:

  • 18 Seems to hang on C file

Merged pull requests:

  • 19 Add some math modules

Fixed cpantesters reports:

You can see the commits at https://github.com/pjcj/Devel--Cover/commits/master

  • Hours worked: 21:20
  • Total hours works on grant: 63:30

Dave Mitchell writes:

As per my grant conditions, here is a report for the June period.

This month saw the final merging of my re_eval branch back into blead, and incorporated in the 5.17.1 release. This consisted of about 125 commits, spread out over a year. The remainder of the month was spent fixing some of the issues thrown up by the new implementation.

Some highlights of the new /(?{})/ implementation are:

Code blocks within patterns are now parsed in the same pass as the surrounding code; in particular it is no longer necessary to have balanced braces: this now works:

/(?{ $x='{' })/

This means that this error message is longer generated:

Sequence (?{...}) not terminated or not {}-balanced in regex

but a new error may be seen:

Sequence (?{...}) not terminated with ')'

In addition, literal code blocks within run-time patterns are only compiled once, at perl compile-time:

for my $p (...) {
    # this 'FOO' block of code is compiled once, at the same time as
    # the surrounding 'for' loop

Lexical variables are now sane as regards scope, recursion and closure behaviour. In particular, /A(?{B})C/ behaves (from a closure viewpoint) exactly like /A/ && do { B } && /C/, while qr/A(?{B})C/ is like sub {/A/ && do { B } && /C/}. So this code now works how you might expect, creating three regexes that match 1,2, and 3:

for my $i (0..2) {
    push @r, qr/^(??{$i})$/;
"1" =~ $r[1]; # matches

The use re 'eval' pragma is now strictly only required for code blocks defined at runtime; in particular in the following, the text of the $r pattern is still interpolated into the new pattern and recompiled, but the individual compiled code-blocks within $r are reused rather than being recompiled, and use re 'eval' isn't needed any more:

my $r = qr/abc(?{....})def/;

Flow control operators no longer crash. Each code block runs in a new dynamic scope, so next etc. will not see any enclosing loops and caller will not see any calling subroutines. return returns a value from the code block, not from any enclosing subroutine.

Over the last month I have averaged 15 hours per week.

As of 2012/06/31: since the beginning of the grant:

120.9 weeks
1300.0 total hours
10.8 average hours per week

There are now 0 hours left on the grant (but the grant has just been extended :-)

Report for period 2012/06/01 to 2012/06/30 inclusive


Effort (HH::MM):

23:50 diagnosing bugs
39:03 fixing bugs
0:00 reviewing other people's bug fixes
0:00 reviewing ticket histories
0:00 review the ticket queue (triage)
62:53 Total

Numbers of tickets closed:

4 tickets closed that have been worked on
0 tickets closed related to bugs that have been fixed
0 tickets closed that were reviewed but not worked on (triage)
4 Total

[actually, an additional 25 tickets were closed, but I forgot to update my automatic log]

Short Detail

8:42 [perl #3634] Capture corruption through self-modying regexp (?{...})
36:41 [perl #34161] METABUG - (?{...}) and (??{...}) regexp issues
4:25 [perl #113670] Bleadperl v5.17.0-451-gf6033a9 breaks SBECK/Date-Manip-6.32.tar.gz
2:00 [perl #113676] Bleadperl v5.17.0-383-g9f14173 breaks PARDUS/File-MimeInfo/File-MimeInfo-0.16.tar.gz
10:05 [perl #113682] Bleadperl v5.17.0-424-gd24ca0c breaks ABIGAIL/Regexp-Common-2011121001.tar.gz
1:00 [perl #113720] perl dumping core in regex et al(mem corruption?)

The Perl Foundation is looking at giving some grants ranging from $500 to $2000 in August 2012.

You don't have to have a large, complex, or lengthy project. You don't even have to be a Perl master or guru. If you have a good idea and the means and ability to accomplish it, we want to hear from you!

Do you have something that could benefit the Perl community but just need that little extra help? Submit a grant proposal until the end of July. You would like to have the money, you have the knowledge, but do not know what do propose? Ask around and you will probably get some ideas.

As a general rule, a properly formatted grant proposal is more likely to be approved if it meets the following criteria

  • It has widespread benefit to the Perl community or a large segment of it.
  • We have reasons to believe that you can accomplish your goals.
  • We can afford it (please, respect the limits or your proposal should be rejected immediately).

To submit a proposal see the guidelines at http://www.perlfoundation.org/how_to_write_a_proposal and TPF GC current rules of operation at http://www.perlfoundation.org/rules_of_operation. Then send your proposal to [email protected] Your submission should be properly formatted accordingly with our POD template.

Proposals will be made available publicly (on this blog) for public discussion, as was done in the previous rounds. If your proposal should not be made public,
please make this clear in your proposal and provide a reason.

Lightning Talks at OSCON

Going to be at OSCON this year? The Perl track will have the traditional lightning talk session along with Larry's State of the Onion. If you've got something to share, you've got 2 days left to submit your lightning talk to give everyone your 5 minutes of wisdom on Thursday evening.

TPF will also have a booth at the expo, so stop by and say hi.

I am pleased to announce that we have extended Dave Mitchell's Fixing Perl 5 Core Bugs grant and Nicholas Clark's Improving Perl 5 grant. This will allow them both to spend another 400 hours improving the Perl 5 core.

I would like to thank the community members who took time to provide feedback on this proposal and all those who have made donations to the Perl 5 Core Maintenance Fund. If you are interested in making a donation to this fund please use our donation system or write to karen at perlfoundation.org.

Jonathan Worthington writes:

This is the final report for my Hague Grant on Meta-model Improvements and Natively Typed Attributes. There was also an intermediate progress report. At the point of the intermediate progress report, a new version of NQP had been delivered. Its objects were implemented in terms of a new meta-object system, developed with the needs of Perl 6 in mind. However, at that point Rakudo had not started to use it. This report covers that part of the work. In this report I will use D1, D2, etc. to refer to Deliverable 1, Deliverable 2, etc. from the original grant application.

Rakudo has evolved and improved enormously during the course of this grant. This is partly thanks to the work of this grant, but also as a consequence of work that has built upon it. In many cases the work done not only meets the goals of the grant but goes far beyond them. Furthermore, the MOP work delivered under this grant is also being studied by those working on the Perl 5 MOP project.

Rakudo today has all of its meta-objects implemented in NQP (D2, D3), using the new meta-object system designed and implemented under this grant, which is often known by the name "6model" (D1). Each meta-object implements the semantics of a kind of type: classes, grammars, roles, subsets, enumerations and so forth. These meta-object implementations are in turn factored in terms of roles, thanks to NQP having been given role support during the course of the grant. The NQP role support has also been used by Patrick Michaud in development of an updated grammar engine, which has in turn resulted in many improvements to Rakudo's grammar support.

The point of a meta-object protocol is not only to enable clean implementation of language features, but also to allow extending the semantics of the language. These extensions can be written in Perl 6 - that is, in the language itself. The ability to extend a language using the language itself is known as meta-circularity. The most useful modules using this ability so far are Grammar::Tracer and Grammar::Debugger, which add debugging support to grammars in pure Perl 6. As well as being a good demonstration of meta-programming in Perl 6, the modules have been genuinely useful to those writing Perl 6 grammars. More examples of meta-programming can be found in my Meta-programming in Perl 6 talk [pdf]. The specification has a section on custom meta-objects also (D5); this will continue to be fleshed out over time.

The grant also called for the implementation of natively typed attributes. This deliverable has been achieved in both NQP and Rakudo (D4). In NQP, they have been used to slim down the Cursor and Match data structures. Furthremore, Rakudo has also gained natively typed lexical variables. As well as being available to user-space programs, they have also been used to optimize various built-ins in the core setting.

Test coverage for the metamodel and other deliverables is available throughout the existing testing frameworks (D6). Testing of 6model and role support is provided in NQP's test suite. Testing of OO programming is performed extensively through the S06, S12, and S14 portions of the roast (spectest) test suite.

During the course of the work, pre-compilation support and BEGIN handling have also been improved greatly. This partly involved the implementation of bounded serialization, which dramatically improved the startup time of Rakudo as well as the load time of pre-compiled modules. It also means that BEGIN time compuations are properly persisted over the compile-time/runtime boundary.

The richer type system support also led to development of an optimizer; this work was presented in a blog post as well as a talk [pdf] given at a couple of workshops. The optimizer supports optimizations such as inlining and compile-time resolution of multiple dispatch. The analysis done by the optimizer can also detect various program errors that will lead to innevitable failure at runtime.

While the optimizer can greatly improve performance in certain situations, the work done in this grant has led to an overall performance increase even without the optimizer. The effect of these improvements on a mandelbrot program was analysed early on in a blog post by Moritz Lenz; further improvements led to running this program 19 times faster than was possible in Rakudo as it was before this grant was started. As another data point, the list/iterator model developed in Rakudo using the primitives provided by this grant is 30x faster than Rakudo previously had, despite being written in Perl 6 rather than PIR.

Also worth mentioning is that by implementing representation polymorphism, the new object system has allowed for greatly improved native call support; it is now possible to pass arrays and structs to C from Perl 6 code using the NativeCall module. The improved native call support has been used in modules such as DBIish, which now supports MySQL, Postgres and Sqlite.

Rakudo is a great deal further forward today than it was before this grant started. While this is of course thanks to the efforts of many contributors, the work done under this grant has been key to the progress than has been made. In many senses, the main success of this grant has not just been in achieving the deliverables themselves, but in the many things that have been built around and on top of them. Critically, Rakudo is in a strong position to continue improving stability, performance and usability, as well as add additional missing features - all important to being useful to a wider range of potential users.

Enrique Nell and Joaquin Ferrero reported:

Project status: https://docs.google.com/spreadsheet/ccc?key=0AkmrG_9Q4x15dC1MNWloU0lyUjhGa2NrdTVTOG5WZVE

CPAN distribution: http://search.cpan.org/~enell/POD2-ES-

Project host: https://github.com/zipf/perldoc-es

During the last month we updated POD2::ES from v5.14.2 to v5.16.0 (translated new segments and updated changed ones), and released to CPAN the updated distribution. Other translated files that still must be reviewed (i.e., that haven't been released to CPAN) are currently being updated to v5.16.0.

POD2::ES v5.16.0.01 included all the files in the previous release, except perlcheat.pod, which had a bug that we reported through perlbug. Discussion in the p5p list suggested that this file would undergo further changes, and since the localized version entails heavy formatting work, we decide to wait until v5.16.1 to include it back in the distribution.

New files added during the v5.16.0 track of POD2::ES:

  • perlbot
  • perlboot
  • perldata
  • perlexperiment
  • perlootut
  • perltodo
  • perltooc
  • perltoot

We also fixed code issues (related to POD2::Base) in ES.pm, the main module, and made a few doc changes.


Here is a brief summary of the final project status for v5.14.2 (June 4):

  • Total documents: 190 (98 in core docs)
  • Total words: 924,435 (619,056 in core docs)
  • % translated: 44.02% (42,30% of core docs)
  • % reviewed: 4.43%

And the current status of our v5.16.0 track:

  • Total documents: 162 (100 in core docs)
  • Total words: 841,604 (597,294 in core docs)
  • % translated: 43.51% (44.74% of core docs)
  • % reviewed: 5.06%


Our new team members, Jennifer Maldonado and Manuel Gómez, started working remotely through a remote server setup deployed in April. New collaborators can start working right away with very little training, since they don't have to install and configure the translation tool (although they can do it if they prefer a local setup; the project wiki contains guidelines for both scenarios.)


We have implemented several changes in the post-processing script:

  • Support for the new naming convention used for Readmes in v5.16
  • Handling of pod files containing the =encoding utf8 command (our translation tool currently generates latin-1 files, so now the post-processing script checks and changes the encoding of the generated file if necessary)
  • Command line switch to add names of people involved in the translation of a particular file to the Translators section at the bottom of the file
  • Hardcoded paths were updated for the new v5.16 project setup.

Future work

For our second month, we plan to do more work on tools and keep translating/reviewing pods while we wait for v5.16.1, which will require another update.

Our immediate TO DO list includes reporting the issues found in POD2::Base and several pod processing issues that should be fixed for localized versions.

For instance, the CPAN page for POD2::ES does not show the description of the translated modules because the POD processor does not find section NAME (it's NOMBRE in Spanish).

The French team fixed this by using NAME/NOM, but we think that a cleaner solution would be better.

Another issue we found is related to the processing of POD links like L, which produces English text: "Supported Platforms" in perlport (for Spanish, "in" should be "en".)

It seems that the *roff-related issues that prevented displaying correctly the extended characters in UTF-8 encoded files using perldoc in Xterminals have been fixed in v5.16.0, so we may switch back to UTF-8.

We are also testing the latest version of the translation tool, OmegaT v2.6, which includes a new feature that allows team work using a single memory.

About TPF

The Perl Foundation - supporting the Perl community since 2000. Find out more at www.perlfoundation.org.

About this Archive

This page is an archive of entries from July 2012 listed from newest to oldest.

June 2012 is the previous archive.

August 2012 is the next archive.

Find recent content on the main index or look in the archives to find all content.


OpenID accepted here Learn more about OpenID
Powered by Movable Type 6.2.2