2011Q2 Grant Proposal: Port pmtools to Perl 6
Sat, 04-Jun-2011 by
Alberto Simões
edit post
<![CDATA[
<p>This quarter, The Perl Foundation Grant Committee received one proposal for grant funding. The proposal is presented bellow, and it is open for discussion.</p>
<h1><a name="port_pmtools_to_perl_6">Port pmtools to Perl 6</a></h1>
<dl>
<dt><strong><a name="name" class="item">Name:</a></strong></dt>
<dd>
<p>Mark Leighton Fisher</p>
</dd>
<dt><strong><a name="amount_requested" class="item">Amount Requested:</a></strong></dt>
<dd>
<p>$1200.00</p>
</dd>
</dl>
<p>
</p>
<h2><a name="synopsis">Synopsis</a></h2>
<p>pmtools, the Perl module tools, are used by Perl 5 programmers every
day to find, use, modify, and fix Perl modules. With over 90 modules
already available for Perl 6, it is time for to have a set of
module tools for Perl 6.</p>
<h2><a name="benefits_to_the_perl_community">Benefits to the Perl Community</a></h2>
<p>Perl's flexibility extends to the locations of its modules,
as out of the box Perl supports both CORE and locally-installed
modules written in both Perl and native code. So without
prior knowledge of the module in question, you cannot
know where the module file is
located, what version it is, whether it was coded
in Perl or another language, whether it was installed locally
or by the core Perl installation process, or anything else
about it. pmtools, the Perl module toolkit, was designed to
remedy that.</p>
<p>pmtools for Perl 5 helps you find:</p>
<ul>
<li>
<p>What modules you have;</p>
</li>
<li>
<p>What version(s) of modules you have;</p>
</li>
<li>
<p>Where those modules are in the filesystem;</p>
</li>
<li>
<p>What is exported by a module;</p>
</li>
<li>
<p>The documentation for a module; and</p>
</li>
<li>
<p>Many other vital facts about your modules (there are
24 tools in the pmtools suite).</p>
</li>
</ul>
<p>It would be possible to rework Perl 5's pmtools to work on Perl 6
modules, thereby avoiding the extra effort to port the pmtools
source to Perl 6.</p>
<p>However, in the long term it would be better yet if the entire
Perl 6 toolchain -- including pmtools -- was written in Perl 6,
both to reduce the number of different languages needed to
understand the Perl 6 toolchain sources as well as with demonstrating
that Perl 6 is a good language for writing such tools.</p>
<p>
</p>
<h2><a name="deliverables">Deliverables</a></h2>
<ul>
<li>
<p>Port the pmtools programs to Perl 6.</p>
</li>
<li>
<p>Port the pmtools installation process to Perl 6.</p>
</li>
<li>
<p>Add smoke-testing (at least).</p>
</li>
<li>
<p>Add any additional necessary documentation.</p>
</li>
</ul>
<p>
</p>
<h2><a name="detailed_list_of_pmtools">Detailed List of pmtools</a></h2>
<ul>
<li>
<p>basepods - list only normal "man-page" style pods</p>
</li>
<li>
<p>faqpods - list only FAQ pods</p>
</li>
<li>
<p>modpods - all module pods, including site_perl ones</p>
</li>
<li>
<p>pfcat - show pods from perlfunc</p>
</li>
<li>
<p>plxload - show what modules a given program loads at compile time</p>
</li>
<li>
<p>pmall - get all installed modules pmdesc descriptions</p>
</li>
<li>
<p>pmcat - cat the module source through your pager</p>
</li>
<li>
<p>pmdesc - get a module description</p>
</li>
<li>
<p>pmdirs - print the perl module path, newline separated</p>
</li>
<li>
<p>pmeth - list a class's methods, recursively</p>
</li>
<li>
<p>pmexp - show a module's exports</p>
</li>
<li>
<p>pmfunc - show a function source code from a module</p>
</li>
<li>
<p>pminst - find what's installed</p>
</li>
<li>
<p>pmload - show what modules a given module loads at compile time</p>
</li>
<li>
<p>pmls - long list the module path</p>
</li>
<li>
<p>pmman - show the module's pod docs</p>
</li>
<li>
<p>pmpath - show the module's full path</p>
</li>
<li>
<p>pmvers - get a module version number</p>
</li>
<li>
<p>podgrep - grep in pods of a file</p>
</li>
<li>
<p>podpath - show full path of pod file</p>
</li>
<li>
<p>pods - list all standard pods and module pods</p>
</li>
<li>
<p>podtoc - list table of contents of a podpage</p>
</li>
<li>
<p>sitepods - list only pods in site_perl directories</p>
</li>
<li>
<p>stdpods - list standard pods, not site_perl ones</p>
</li>
</ul>
<p>
</p>
<h2><a name="project_details">Project Details</a></h2>
<p>pmtools are seen in Perl 5 community as a valuable addition to the
developer's toolkit, to the point that they were day 16 in the 2009
Perl Advent Calendar and a part of the Fedora Linux distribution.
Created originally by Tom Christiansen, I took over
pmtools in 2006 when link decay had removed them from the Web.
I have released 2 additional versions of pmtools (1.01, a
bugfix version, and 1.10, which dual-licensed the pmtools under GNU
and Artistic licenses).</p>
<p>I started work on a port of pmtools to Perl 6 in 2007, releasing
config-Perl6 as a module needed for the Pugs implementation of the
Perl 6 pmtools, but the combination of the immaturity of the Perl 6
toolchain and my own time commitments have made progress very slow.
Rakudo Perl solves the Perl 6 toolchain immaturity problem for me,
as any obstacles I have found thus far with Rakudo have been
easily overcome (I have only run into syntax quirks so far).
A grant for porting pmtools to Perl 6 would permit me to put my full
consulting time into this project rather than running this port as a
hobby project, thereby solving the time commitment problem.</p>
<p>The biggest project risk are the 2 most difficult programs to port
to Perl 6:</p>
<ul>
<li>
<p>pmload - show what modules a given module loads at compile time;</p>
</li>
<li>
<p>plxload - show what modules a given program loads at compile time;</p>
</li>
</ul>
<p>as porting pmload and plxload requires digging into the internals of
the Perl 6 module loading process, while all of the other pmtools
require at most following the same rules for module loading as
Perl 6 itself (and the POD pmtools do not even require that). The
other pmtools would still be useful even if one or both of these
programs remained unported to Perl 6 during the project timeframe.</p>
<p>Another project risk is that no Perl 6 implementation is finished
like a Perl 5 implementation is finished. There may be difficulties I
cannot foresee in porting particular pmtools because of bugs and
missing features in the Perl 6 toolchain. These difficulties could
be classified as "known unknowns", as I know before starting
this project that the development of an initial full release of
Perl 6 is still underway.</p>
<p>
</p>
<h2><a name="inch_stones">Inch-stones</a></h2>
<p>The logical "inch-stone" for this project is a completed pmtools
program, which gives me 24 inch-stones for this project (an adequate
level of granularity, I think).</p>
<p>Each week I will create a new distribution of pmtools-perl6 from the
Perl 6 pmtools completed so far and place it at modules.perl6.org
and in my Sourceforge project pmtools-perl6. This means that the
Perl 6 community will start seeing immediate value from the project.</p>
<p>
</p>
<h2><a name="project_schedule">Project Schedule</a></h2>
<p>The project should take 2 months (as you can see from the
schedule below). I can start on it as soon as the grant is
approved.</p>
<ul>
<li>
<p>Week 1: pmpath, pmvers, pmdirs, pmls, and installation process.</p>
</li>
<li>
<p>Week 2: pminst, pmall, and verify that Audrey Tang's Pugs version of GetOpt::Std works under Rakudo, fixing it if necessary.</p>
</li>
<li>
<p>Week 3: pmeth, pmexp, and pmfunc.</p>
</li>
<li>
<p>Week 4: pmman, pmcat, pmdesc, and podtoc.</p>
</li>
<li>
<p>Week 6: basepods, podgrep, and podpath.</p>
</li>
<li>
<p>Week 7: faqpods, modpods, pods, sitepods, and stdpods.</p>
</li>
<li>
<p>Week 8: pfcat, pmload, and plxload.</p>
</li>
</ul>
<p>
</p>
<h2><a name="completeness_criteria">Completeness Criteria</a></h2>
<p>Completeness criteria are whether each program gives the same results
as the equivalent Perl 5 pmtools program, given the same inputs.
(I do not anticipate making major, externally-visible changes to how
each program functions.)</p>
<p>The documentation for each program will be an updated
copy of the documentation for its corresponding Perl 5 pmtool
program.</p>
<p>
</p>
<h2><a name="bio">Bio</a></h2>
<p>I am Mark Leighton Fisher, long-time Perl user (dating back to
Perl 4 and my initial implementation of the TCE Corporate Technical
Memory in the 1990's), Perl CORE and CPAN author with several patches
to my credit (including the "one-at-a-time" string-hashing algorithm
that was used for Perl 5 hashes), and maintainer and popularizer of
the Perl 5 pmtools since 2006, of which I have released 3 versions on
CPAN (initial CPAN version, bugfix version, and the dual-licensed
(Artistic and GNU) update version for inclusion with Fedora Linux).</p>
<ul>
<li>
<p><a href="http://search.cpan.org/~mlfisher/">http://search.cpan.org/~mlfisher/</a> - Some of my Perl 5 source code.</p>
</li>
<li>
<p><a href="http://sourceforge.net/projects/pmtools-perl6/files/pmtools-perl6/">http://sourceforge.net/projects/pmtools-perl6/files/pmtools-perl6/</a> - Some of my Perl 6 source code.</p>
</li>
</ul>
<p>I have also written articles for Perl.com and The Perl Review.</p>
]]>
Comments (5)
What tools will you use for parsing Perl 6 POD? Or do you plan to write your own? What good is a pfcat when there's no perlfunc document for Perl 6?
What tools will you use to obtain export lists from a module? Are you going to handle versioned module dependencies? Can you elaborate on the meaning of site_perl in the context of Perl 6?
I like the idea, but the proposal doesn't seem to reflect fundamental differences between Perl 5 and Perl 6.
1) So the Perl 6 pmtools remain a lightweight toolset, I plan on writing the POD parsing code myself. pmtools was never designed to have a presentation-quality POD parser -- it was designed to quickly and easily get the relevant POD information from the Perl and POD source files. Though someone could prove me wrong, Perl 6 POD (from my looks at it) was designed to be just a step up from Perl 5 POD, not a huge departure.
2) pfcat will be useful when there is Perl 6 perlfunc, as we Perl 6 programmers will eventually need.
3) Though I don't remember the syntax off-hand, I think you can just ask a Perl 6 modules for its exports, rather than parsing the module code as the Perl 5 pmtools must. (And we have better support for parsing Perl 6 than we have for parsing Perl 5.)
4) My long-term plan is to support versioned module dependencies, though last I looked no one had supported versioned modules in their module code yet.
5) Assuming that no one comes up with a better mechanism, site_perl for Perl 6 will contain the Perl 6 modules that are installed after the initial Perl 6 installation. Have you heard anything different about plans for site_perl in Perl 6?
6) I don't see huge differences in Perl 6 vs. Perl 5 except for features designed for better programming-in-the-large -- the standardized object model, flexible multi-method dispatch, built-in Perl 6 parsing support, etc. I am curious to know where you think Perl 6 is different enough from Perl 5 that the general pmtools approach -- small, lightweight tools that do just enough to get developers the information they need -- will fail for a Perl 6 pmtools.
7) Though I did not state this explicitly in my grant proposal, I am aware that Perl 6 is different from Perl 5, so the pmtools porting process is not just like moving a C program from a 32-bit little-endian Unix machine to a 32-bit little-endian Linux machine -- there may be Perl 6 differences from Perl 5 that require a different approach to the particular Perl module development problem than taken by the corresponding Perl 5 pmtool program. But the pmtools are a "gist extraction machine" as it were -- designed to get developers the module information they need without a lot of fuss (or intense, complex coding efforts on the part of the pmtools developer). If you want tools suitable for building presentations about multi-billion dollar (euro, etc.) projects, you may want to look elsewhere. But if you want to check a current module's version, find a module's code to check how it handles an edge case, etc. then the pmtools are a good fit.
currently, I'm -1 to any grant on Perl6 that is not related to the core improvement of rakudo or parrot.
This proposal seems a tad premature. pmtools works in Perl 5 because it builds on the existing module infrastructure. There isn't much of a module infrastructure for Perl 6 yet.
Rather than copy a set of tools that were designed to solve a set of Perl 5 problems in a Perl 5 universe, better would be to let the Perl 6 universe evolve its own tools that solve whatever problems come up as it evolves its own module system. (IMHO, of course)
I'm curious as to what you would like to see in Perl 6 module infrastructure, as we already have 90+ Perl 6 modules at http://modules.perl6.org/.
As Perl 6 developers, we are the agents of evolution -- Perl 6 module tools will not evolve without someone making them evolve. I'm trying to get ahead of the game here, so we do better than we did with Perl 5 (when (I think) there were over 1000 modules in CPAN when Tom Christiansen wrote the first version of pmtools).
Perl 6 is an advance (a large one) over Perl 5, but we will still store our source code in files, our source will still use explicit versioning, we will still document with POD, etc. -- all of which bring their own issues to the table. Rather than forcing a huge infrastructure on each and every module so you could just ask that module for all of its information, it would be better to have a set of loosely-coupled tools to help developers find the information they need about modules -- hence pmtools.
Perl (5 or 6) is not a huge, tightly-coupled system like a LISP or a Smalltalk. Perl is closer to the Unix/Linux command-line tool set or much of the Web infrastructure, with numerous tools communicating with each other over a standard set of protocols. Creating pmtools as a separate set of tools is closer to that style of architecture.
I think it is worth reflecting on Gall's Law here -- "A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system." pmtools is a simple system of tools for dealing module metadata.
