July 2013 Archives

Nicholas Clark writes:

As per my grant conditions, here is a report for the June period.

The nice thing about standards is that you have so many to choose from (Andrew S. Tanenbaum).

I guess the same can be said about build systems.

So the structural intent of the build is

1. Permit the user to choose configuration options
2. Build the package (which may take some time, and shouldn't need user intervention)
3. Test the package, and collate all test results into one report at the end (an excuse for a second tea break)
4. Install the package (which probably runs with elevated privileges)

As well as trying to avoid a long period where a human needs to babysit the build in case it stops to ask a question, this approach also has the benefit that you find out by the end of configuration what extensions the build stage should be producing. Or, more importantly (compared with at least one other similar language), you don't need to wait until the end of the build run to discover that an extension you really needed isn't built, and then have to iterate the entire configure & build steps until you figure out the correct form of rubber chicken sacrifice to make it all work.

Of course, the problem is that for step 1 one can't assume you have a copy of Perl already (because how did it get built?) so the configuration system has to run using native tools. And the more platforms the package is ported to, the more variations of native tools you have.

So, on *nix and VMS, where the OS, architecture and even the make utility will vary, the configuration script figures out which extensions are shipped by scanning the file system, because even the Makefile has to be programmatically generated to cope with platform quirks. On Win32 variations are a lot less, so it's viable to ship a pair of Makefiles which between them cover all the common make variants. Hence on Win32 configuration is implemented by changing options in the appropriate Makefile, and the build determines which extensions are wanted by combining those options with a scan done by the (uninstalled) FindExt module.

So that's a Perl module right? Which means that we can test it in a platform-independent way. Which turned out to be useful back in 2009 when I was working out how to move modules to cpan/ dist/ and ext/ as part of the big rearranging to make dual life a lot simpler, as I could mostly verify that my changes were going to work on Win32 without having any direct access to a Win32 system to test it. The tests written for that purpose were robust enough that they were moved to t/porting and run as standard, which verifies that the logic in FindExt is consistent with that of Configure.

However we weren't able to test everything. We couldn't correctly test the list of static extensions due to various problems, and list of dynamically built extensions failed match due to 2 discrepancies between Configure logic and FindExt.

Nicholas Clark writes:

As per my grant conditions, here is a report for the May period.

This month I simplified part of the implementation of pack and unpack, removing about 130 lines of code, and reducing the object code size by about 2K. The only casualty was support for mixed-endian systems. Sorry, PDP-11 users.

In the medal stakes for "risk to sanity", the implementation of pack and unpack are strong contestants, although only for bronze. (They're fighting it out with the implementation of sprintf. Compared with sprintf, they have the advantage that they've been seen as sufficiently dangerous to nearby code that they've been quarantined into their own file. The regular expression engine gets silver, and the reigning champion and gold medal holder remains the parser itself.)

pack/unpack is one of those pieces of code which has grown organically as features were added, and so has become too large to easily comprehend the big picture. In particular, it's hard to look at it at more than one level at once, and hence see where it could be profitably refactored.

The main code of pp_pack and pp_unpack are large switch statements, which are used to dispatch a single template code, and perform the correct sized and typed read or write of bytes to or from the string. There seems to be a lot of repetition in there, but it's more at a "template" level than a code level, because the types of the variables in each case differ. Pretty much all the things that are the same size and could be merged have been merged. There's no obvious way to squeeze it further with this design.

However, one thing did strike me as odd - a pattern of repeated macros in the unpack code, particularly in all the cases that dealt with the various sizes and signedness of integers. They exist due to the order that the code was written and then enhanced. The original code (dating from Perl 3 era) read numbers as chunks of memory. It copied bytes to/from the (possibly unaligned) input string to a local variable of the correct type. Much much later Marcus Holland-Moritz added code to provide the pack modifiers "<" and ">" (for endianness) and implemented the endian swapping they require, by endian swapping the values in the integer variables. Subsequently Ton Hospel added code to cope with the unpack input string having been internally transcoded to UTF-8, by having the copy transcode back if needed. These two later additions are respectively the second and first macros.

We have received the following Hague Grant application from diakopter.
Before we vote on this proposal we would like to have a period of community consultation for 10 days. Please leave feedback in the comments or if you prefer send email with your comments to karen at perlfoundation.org.

Applicant's Name: diakopter

Project Title

Wrap up Perl 5 Interop for Rakudo Perl 6 on MoarVM (2 months)

Synopsis

Embed Perl 5 in Rakudo's implementation on one of its two newest backends, MoarVM, in order to enable seamless two-way interop between the Perl 6 environment and the Perl 5 environment (9 weeks).

Benefits to Perl 6 Development

Rakudo Perl 6 currently has limited access to Perl 5 modules and CPAN distributions via Blizkost, but only on the Parrot VM. Tobias Leich is building rakudo-p5/v5 (use v5;), which is a reimplementation of Perl 5 in Perl 6, but that effort will be able to support only a small subset of what's on CPAN. (However, see below on how it could help in the future.) Bringing all of CPAN to Rakudo via embedding the Perl 5 interpreter would open up entirely new worlds of opportunities and additional Perl 6 (and Perl 5!) adoption and maintenance paths.

Deliverables & Project Details

We expect Rakudo to be basically done being ported to the JVM by August, and then also to MoarVM not too many weeks after that. I can get much of the coding done before Rakudo is bootstrapped on MoarVM, but nearly all of the writing tests in Perl 6 would occur in the third month. As far as coding goes, I've already sketched out around 40% of the code for the Perl 5 interop.

Long-term, there are many potential Perl 5 Interop deliverables, but to start, it is an extension to MoarVM and Rakudo p6, either distributed separately or optionally built-in (or both), that embeds p5 libperl to enable calling p5 code (interacting with p5 objects) from p6, and vice versa, for the typical use case of using p5 CPAN distributions in p6 code.

Feature details/schedule:

1. finalize MoarVM's dynamic extension interface protocols & standards (using DynCall - design already pushed to the extops branch). Note: dependency on importing DynCall into MoarVM (from NQP proper). See http://goo.gl/W5mI0 for the design
2. Skeleton extension "mvmp5" that installs the custom op for p5 interop and installs the custom REPRs into MoarVM/6model (for the p5 object wrappers and others)
3. Custom p5 REPRs - MVMP5Interpreter, MVMP5Val, MVMP5Method, MVMP5Package
4. Custom p5 op - mvmp5::code(string), which takes p5 code and creates an invokable MVMP5Val (wrapping a CV ref)
5. Appropriate handling (symbol synchronization) of outermost "use" statements to mvmp5::code (not testable until Rakudo (or NQP) is bootstrapped on MoarVM, because the use case of the feature is compile-time from the HLL's parser)
6. Custom MetaObject implementation (mostly in NQP) for MVMP5Val and MVMP5Package objects (mostly for find_method)
7. Special handling of p6 type objects as represented in p5, p5 packages in p6
8. Special handling of HLL-config-ish things of p5 objects in p6, and likewise special handling of override.pm and TIE interfaces of p6 objects in p5
9. Wiring up the two VMs' GCs by an intermediate references table with refcounts for p6 pointers from p5, and double-increment refcount of p5 pointers from p6, so p6 can hold a reference while also being able to control exactly when the final reference is decremented
10. calling p5 code (invokable MVMP5Vals that wrap CVs) from p6 (including arg flattening & coercion)
11. Method calls on p5 objects from p6 (use of MVMP5Method intermediate invokables)
12. Method calls on p6 objects from p5 (with the optimization of installing stubs into the stashes, a la Moose class finalization)
13. Integration of CallAsOp module to avoid nested p5 runloops (enabling co-recursive cross VM callbacks in a stackless manner)
14. p5 wrappers of p6 objects using blessed/tied/autoload SVrefs that all refer to the same dummy SV; indexes to the MVMObjects representing the reference table offsets of each's origin MVMP5Interpreter and referent MVMObject are embedded in another magic on the SV
15. Marshaling of value types bidirectionally on args to the VMs (P6int/num MVMObjects & IV/NV, e.g.) cross-invocations
16. Coercion of p5 args to mvm ints/nums/strs registers. These are not [yet] listed in implementation order.

The design of the p5 interop scheme has been thoroughly discussed with Larry Wall, Jonathan Worthington, and Nicholas Clark. A prominent p5p-er offered to help drive any changes needed to p5 to support it, but based on my current design and understanding, no changes will be necessary to support 5.8.3 and later. See http://goo.gl/WLJUQ for one iteration of the working p5interop design doc.

Note: we also considered the JVM while designing this scheme, and nearly all of the design of the MoarVM integration can be directly translated to work on the JVM, even the stackless co-recursive callbacks, since Stefan recently added support for stack saving and resuming for implementation continuations for NQP/rakudo on the JVM. It would probably be another couple of weeks work to port it to the JVM also. Note also that this would provide a more complete bridge from p5 to the JVM via p6 (more complete than Larry's prior work that shipped with p5 for a long time and other Java interop projects).

Project Details

(also see deliverables.) Long term, goals for the p5 interop include well- "productized" (though not commercialized!) tools to package p6 modules as p5 distributions (via MoarVM/Rakudo p6 provided as an extension to p5), the converse (p5 extensions installable to the local p5 module trees via the p6 library management tools), programmatic dual (but integrated) packaging of p6 code for deployment/distribution to both p5 and p6 installations from the CPAN system. The potential exists to use Tobias Leich's p5 grammar & parts of his p5 re-implementation to better facilitate seamless p5 code inline with p6 code as S01 describes, but the modest goals of this project don't include that functionality; this is mainly for loading p5 CPAN modules and scripts into the p6 runtime and accessing their exposed interaction points. The current design also leaves room for p5 ithreads (but not forks.pm) to be supported in the future, but isn't entirely in scope for this proposal.

Project Schedule

I expect the p5 interop to take up to 9 weeks, commencing anytime.

Report Schedule

Progress reports will be mentioned weekly to the #perl6 and #moarvm channels on Freenode IRC, and as such will be published in the logs at irc.perl6.org; certain major milestones of feature availability will get blog posts on the planet Perl 6 syndication feed.

Public Repository

MoarVM's git repository is at github.com/MoarVM, and changelogs for these contributions will mostly exist in the commit logs, as there likely won't be "releases" of MoarVM separately from NQP itself (as MoarVM is fundamentally an NQP backend).

Grant Deliverables ownership/copyright and License Information

MoarVM is published under the Artistic License 2.0, as are these sub-projects. The Perl Foundation will retain copyright of these initial contributions, but not the software as a whole, because patches and fixes to the software won't usually be able to have their copyright assigned to TPF.

Bio

I'm a co-founder of MoarVM, the Perl6-specialized virtual machine. I designed and implemented (or will implement) the strings/IO subsystems, HLL locks and atomic operations, the bytecode validator and disassembler, the QAST->MAST compiler, both VM-internal and REPR-level hashes, the regex compiler, coordination of ending threads and portions of the GC parallelization and work-stealing system, the native and HLL extension system/interface, the threadsafe (and lock-free!) hash and array REPRs and their auto-upgrading from their speedier single-threaded editions, the NFG implementation, the Unicode Character Database character properties and case changes compilation/runtime, and finally the Perl 5 interop scheme.

Amount Requested

USD$10k for 2 months at $5k/month.

Jeffrey Ryan Thalhammer reported:

This past May, The Perl Foundation awarded a grant to fund development of a couple features in Pinto. Pinto is a robust tool for curating a private repository of CPAN modules, so you can build your application with the right modules every time. This is my first progress report on that work.

The grant proposal covered two key features. I have a tendency to pick the low-hanging fruit first, so I started with the easier of the two.

Pinto-0.087_03 has been released to CPAN and the diff command now supports both revision IDs and stack names as arguments. The IDs can be abbreviated to uniqueness (just like git). When a stack name is used, the head revision of that stack is assumed. Complete documentation is in the manual for the diff command.

This is a development release for now, and I will make a production release after the CPAN Testers give me some passing reports. I will also add support for git-like notation for specifying an offset from the head. For example, foo^ would mean one revision prior to the head of stack foo. Eventually, I may add support for specifying dates too, but that is more distant.

The other feature covered by the grant is merging of stacks. This is an extremely powerful feature, but I'm not entirely sure how it should behave. Their has been some discussion about it on GitHub. So if you have an opinion, feel free to express it at https://github.com/thaljef/Pinto/issues/52

I will publish another progress report in about a month. Many thanks to TPF and the Perl community at large for helping to support this project.

Dr. Nicholas Clark has requested an extension of $20,000 for his Improving Perl 5 grant. This grant started in September 2011 and has been running successfully since then. The requested extension would allow Nicholas to devote another 400 hours to the project. The funds for this extension would come from the Perl 5 Core Maintenance Fund.

As well as weekly reports posted on the p5p mailing list Nicholas provides detailed monthly reports, the most recent of these can be found in the following blog posts:

April 2013
March 2013
February 2013

Before we make a decision on this extension we would like to have a period of community consultation that will last for seven days. Please leave feedback in the comments or, if you prefer, email your comments to karen at perlfoundation.org.

As you might have noticed, TPF has been granting money for some big tasks, like funding Nicholas Clark or Dave Mitchel's work on Perl 5. Nevertheless, TPF has a Grants Committee with its own budget, to give grants for smaller projects, ranging from $500 to $3000.

With this amount we do not expect to fund full-time work, but instead, use it as an incentive to complete some specific task. Therefore, you don't have to have a large, complex, or lengthy project. You don't even have to be a Perl master or guru. If you have a good idea and the means and ability to accomplish it, we want to hear from you!

If you have something that could benefit the Perl community but just need that little extra help, submit a grant proposal until the end of July. You would like to have the money, you have the knowledge, but do not know what do propose? Ask around and you will probably get some ideas. Some Perl pumpkins might post some ideas as comments on this post.

Check procedure details at http://www.perlfoundation.org/how_to_write_a_proposal

I am pleased to announce that The Perl Foundation (TPF) has accepted Tony Cook's grant application for Maintaining Perl 5. As always the application was considered after a public consultation and feedback period. I would like to thank everyone
who took the time to view, assess, and provide feedback on the application.

This grant was awarded from money previously raised for the Perl 5 Core Maintenance Fund, an initiative maintained by TPF to manage, evolve, and support work on the Perl 5 core.

This fund relies entirely on the generosity of our sponsors, community donations, and internal fundraising. Without these funds we would not be able to support this important work. If you would like to contribute to this fund, or know of an individual or organisation who would like to contribute then please direct them to our donation page. Donations of any size are gratefully accepted and are a valuable addition.