2011Q2 Grant Proposal: Port pmtools to Perl 6

5 Comments

This quarter, The Perl Foundation Grant Committee received one proposal for grant funding. The proposal is presented bellow, and it is open for discussion.

Port pmtools to Perl 6

Name:

Mark Leighton Fisher

Amount Requested:

$1200.00

Synopsis

pmtools, the Perl module tools, are used by Perl 5 programmers every day to find, use, modify, and fix Perl modules. With over 90 modules already available for Perl 6, it is time for to have a set of module tools for Perl 6.

Benefits to the Perl Community

Perl's flexibility extends to the locations of its modules, as out of the box Perl supports both CORE and locally-installed modules written in both Perl and native code. So without prior knowledge of the module in question, you cannot know where the module file is located, what version it is, whether it was coded in Perl or another language, whether it was installed locally or by the core Perl installation process, or anything else about it. pmtools, the Perl module toolkit, was designed to remedy that.

pmtools for Perl 5 helps you find:

  • What modules you have;

  • What version(s) of modules you have;

  • Where those modules are in the filesystem;

  • What is exported by a module;

  • The documentation for a module; and

  • Many other vital facts about your modules (there are 24 tools in the pmtools suite).

It would be possible to rework Perl 5's pmtools to work on Perl 6 modules, thereby avoiding the extra effort to port the pmtools source to Perl 6.

However, in the long term it would be better yet if the entire Perl 6 toolchain -- including pmtools -- was written in Perl 6, both to reduce the number of different languages needed to understand the Perl 6 toolchain sources as well as with demonstrating that Perl 6 is a good language for writing such tools.

Deliverables

  • Port the pmtools programs to Perl 6.

  • Port the pmtools installation process to Perl 6.

  • Add smoke-testing (at least).

  • Add any additional necessary documentation.

Detailed List of pmtools

  • basepods - list only normal "man-page" style pods

  • faqpods - list only FAQ pods

  • modpods - all module pods, including site_perl ones

  • pfcat - show pods from perlfunc

  • plxload - show what modules a given program loads at compile time

  • pmall - get all installed modules pmdesc descriptions

  • pmcat - cat the module source through your pager

  • pmdesc - get a module description

  • pmdirs - print the perl module path, newline separated

  • pmeth - list a class's methods, recursively

  • pmexp - show a module's exports

  • pmfunc - show a function source code from a module

  • pminst - find what's installed

  • pmload - show what modules a given module loads at compile time

  • pmls - long list the module path

  • pmman - show the module's pod docs

  • pmpath - show the module's full path

  • pmvers - get a module version number

  • podgrep - grep in pods of a file

  • podpath - show full path of pod file

  • pods - list all standard pods and module pods

  • podtoc - list table of contents of a podpage

  • sitepods - list only pods in site_perl directories

  • stdpods - list standard pods, not site_perl ones

Project Details

pmtools are seen in Perl 5 community as a valuable addition to the developer's toolkit, to the point that they were day 16 in the 2009 Perl Advent Calendar and a part of the Fedora Linux distribution. Created originally by Tom Christiansen, I took over pmtools in 2006 when link decay had removed them from the Web. I have released 2 additional versions of pmtools (1.01, a bugfix version, and 1.10, which dual-licensed the pmtools under GNU and Artistic licenses).

I started work on a port of pmtools to Perl 6 in 2007, releasing config-Perl6 as a module needed for the Pugs implementation of the Perl 6 pmtools, but the combination of the immaturity of the Perl 6 toolchain and my own time commitments have made progress very slow. Rakudo Perl solves the Perl 6 toolchain immaturity problem for me, as any obstacles I have found thus far with Rakudo have been easily overcome (I have only run into syntax quirks so far). A grant for porting pmtools to Perl 6 would permit me to put my full consulting time into this project rather than running this port as a hobby project, thereby solving the time commitment problem.

The biggest project risk are the 2 most difficult programs to port to Perl 6:

  • pmload - show what modules a given module loads at compile time;

  • plxload - show what modules a given program loads at compile time;

as porting pmload and plxload requires digging into the internals of the Perl 6 module loading process, while all of the other pmtools require at most following the same rules for module loading as Perl 6 itself (and the POD pmtools do not even require that). The other pmtools would still be useful even if one or both of these programs remained unported to Perl 6 during the project timeframe.

Another project risk is that no Perl 6 implementation is finished like a Perl 5 implementation is finished. There may be difficulties I cannot foresee in porting particular pmtools because of bugs and missing features in the Perl 6 toolchain. These difficulties could be classified as "known unknowns", as I know before starting this project that the development of an initial full release of Perl 6 is still underway.

Inch-stones

The logical "inch-stone" for this project is a completed pmtools program, which gives me 24 inch-stones for this project (an adequate level of granularity, I think).

Each week I will create a new distribution of pmtools-perl6 from the Perl 6 pmtools completed so far and place it at modules.perl6.org and in my Sourceforge project pmtools-perl6. This means that the Perl 6 community will start seeing immediate value from the project.

Project Schedule

The project should take 2 months (as you can see from the schedule below). I can start on it as soon as the grant is approved.

  • Week 1: pmpath, pmvers, pmdirs, pmls, and installation process.

  • Week 2: pminst, pmall, and verify that Audrey Tang's Pugs version of GetOpt::Std works under Rakudo, fixing it if necessary.

  • Week 3: pmeth, pmexp, and pmfunc.

  • Week 4: pmman, pmcat, pmdesc, and podtoc.

  • Week 6: basepods, podgrep, and podpath.

  • Week 7: faqpods, modpods, pods, sitepods, and stdpods.

  • Week 8: pfcat, pmload, and plxload.

Completeness Criteria

Completeness criteria are whether each program gives the same results as the equivalent Perl 5 pmtools program, given the same inputs. (I do not anticipate making major, externally-visible changes to how each program functions.)

The documentation for each program will be an updated copy of the documentation for its corresponding Perl 5 pmtool program.

Bio

I am Mark Leighton Fisher, long-time Perl user (dating back to Perl 4 and my initial implementation of the TCE Corporate Technical Memory in the 1990's), Perl CORE and CPAN author with several patches to my credit (including the "one-at-a-time" string-hashing algorithm that was used for Perl 5 hashes), and maintainer and popularizer of the Perl 5 pmtools since 2006, of which I have released 3 versions on CPAN (initial CPAN version, bugfix version, and the dual-licensed (Artistic and GNU) update version for inclusion with Fedora Linux).

I have also written articles for Perl.com and The Perl Review.

5 Comments

What tools will you use for parsing Perl 6 POD? Or do you plan to write your own? What good is a pfcat when there's no perlfunc document for Perl 6?
What tools will you use to obtain export lists from a module? Are you going to handle versioned module dependencies? Can you elaborate on the meaning of site_perl in the context of Perl 6?

I like the idea, but the proposal doesn't seem to reflect fundamental differences between Perl 5 and Perl 6.

1) So the Perl 6 pmtools remain a lightweight toolset, I plan on writing the POD parsing code myself. pmtools was never designed to have a presentation-quality POD parser -- it was designed to quickly and easily get the relevant POD information from the Perl and POD source files. Though someone could prove me wrong, Perl 6 POD (from my looks at it) was designed to be just a step up from Perl 5 POD, not a huge departure.

2) pfcat will be useful when there is Perl 6 perlfunc, as we Perl 6 programmers will eventually need.

3) Though I don't remember the syntax off-hand, I think you can just ask a Perl 6 modules for its exports, rather than parsing the module code as the Perl 5 pmtools must. (And we have better support for parsing Perl 6 than we have for parsing Perl 5.)

4) My long-term plan is to support versioned module dependencies, though last I looked no one had supported versioned modules in their module code yet.

5) Assuming that no one comes up with a better mechanism, site_perl for Perl 6 will contain the Perl 6 modules that are installed after the initial Perl 6 installation. Have you heard anything different about plans for site_perl in Perl 6?

6) I don't see huge differences in Perl 6 vs. Perl 5 except for features designed for better programming-in-the-large -- the standardized object model, flexible multi-method dispatch, built-in Perl 6 parsing support, etc. I am curious to know where you think Perl 6 is different enough from Perl 5 that the general pmtools approach -- small, lightweight tools that do just enough to get developers the information they need -- will fail for a Perl 6 pmtools.

7) Though I did not state this explicitly in my grant proposal, I am aware that Perl 6 is different from Perl 5, so the pmtools porting process is not just like moving a C program from a 32-bit little-endian Unix machine to a 32-bit little-endian Linux machine -- there may be Perl 6 differences from Perl 5 that require a different approach to the particular Perl module development problem than taken by the corresponding Perl 5 pmtool program. But the pmtools are a "gist extraction machine" as it were -- designed to get developers the module information they need without a lot of fuss (or intense, complex coding efforts on the part of the pmtools developer). If you want tools suitable for building presentations about multi-billion dollar (euro, etc.) projects, you may want to look elsewhere. But if you want to check a current module's version, find a module's code to check how it handles an edge case, etc. then the pmtools are a good fit.

currently, I'm -1 to any grant on Perl6 that is not related to the core improvement of rakudo or parrot.

This proposal seems a tad premature. pmtools works in Perl 5 because it builds on the existing module infrastructure. There isn't much of a module infrastructure for Perl 6 yet.

Rather than copy a set of tools that were designed to solve a set of Perl 5 problems in a Perl 5 universe, better would be to let the Perl 6 universe evolve its own tools that solve whatever problems come up as it evolves its own module system. (IMHO, of course)

I'm curious as to what you would like to see in Perl 6 module infrastructure, as we already have 90+ Perl 6 modules at http://modules.perl6.org/.

As Perl 6 developers, we are the agents of evolution -- Perl 6 module tools will not evolve without someone making them evolve. I'm trying to get ahead of the game here, so we do better than we did with Perl 5 (when (I think) there were over 1000 modules in CPAN when Tom Christiansen wrote the first version of pmtools).

Perl 6 is an advance (a large one) over Perl 5, but we will still store our source code in files, our source will still use explicit versioning, we will still document with POD, etc. -- all of which bring their own issues to the table. Rather than forcing a huge infrastructure on each and every module so you could just ask that module for all of its information, it would be better to have a set of loosely-coupled tools to help developers find the information they need about modules -- hence pmtools.

Perl (5 or 6) is not a huge, tightly-coupled system like a LISP or a Smalltalk. Perl is closer to the Unix/Linux command-line tool set or much of the Web infrastructure, with numerous tools communicating with each other over a standard set of protocols. Creating pmtools as a separate set of tools is closer to that style of architecture.

I think it is worth reflecting on Gall's Law here -- "A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system." pmtools is a simple system of tools for dealing module metadata.

Leave a comment

About TPF

The Perl Foundation - supporting the Perl community since 2000. Find out more at www.perlfoundation.org.

Recent Comments

  • Mark Leighton Fisher: I'm curious as to what you would like to see read more
  • perlpilot: This proposal seems a tad premature. pmtools works in Perl read more
  • JimmyZ: currently, I'm -1 to any grant on Perl6 that is read more
  • Mark Leighton Fisher: 1) So the Perl 6 pmtools remain a lightweight toolset, read more
  • perlgeek.de: What tools will you use for parsing Perl 6 POD? read more

About this Entry

This page contains a single entry by Alberto Simões published on June 4, 2011 1:21 PM.

Parrot Embed Grant - Report #6 was the previous entry in this blog.

Fixing Perl5 Core Bugs: Report for Month 15 is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

OpenID accepted here Learn more about OpenID
Powered by Movable Type 4.38