2009Q1 Grant Proposal: Interactive Perl Interpreter
Fri, 06-Feb-2009 by
Alberto Simões
edit post
<![CDATA[
<dl>
<dt><strong>Name:</strong>
<dd>
<p>Erik Osheim</p>
</dd>
</li>
<dt><strong>Email:</strong>
<dd>
<p>[hidden email]</p>
</dd>
</li>
<dt><strong>Amount Requested:</strong>
<dd>
<p>$1000</p>
</dd>
</li>
</dl>
<p>
</p>
<h2>Synopsis</h2>
<p>This project aims to build an interactive interpreter for Perl. The goals of
this project are to allow pasted Perl code to run line-by-line as it would be
in a Perl script, provide tab completion and introspection at least equal to
that provided by the Perl debugger, to provide documentation and packaging, and
lastly (and optionally) a package to help with introspection.</p>
<h2>Benefits to the Perl Community</h2>
<p>Interpreted languages like Scheme, Python and Ruby all have interactive
interpreters to help people begin programming, learn tricks and eventually
delve deeper into the lower-level details of the language. Despite having a
robust debugger, Perl has never had precisely such a tool. Several interpreters
exist online to try to fill this niche, but none of them address the main
problems that the perl debugger faces (multi-line statements and blocks fail,
<code>my()</code> does not work as expected).</p>
<p>An interactive interpreter would serve an important pedagogical role. Tutorials
are easier to follow with "bash-like" interactive line-editing, history, and
tab-completion. "Learn by doing" becomes easier when programming interactively,
when errors in syntax or semantics become more immediate.</p>
<p>This project also answers criticisms from other language communities that Perl
needs (or at least could use) an interactive interpreter. Even experienced Perl
programmers would benefit from being able to test and refine particularly
interesting constructions in an interactive shell.</p>
<p>
</p>
<h2>Deliverables</h2>
<p>The program and manual page are mandatory deliverables for the project.
Depending on feedback from the community and usefulness, the IPerl packages may
also be included (possibly with different names). The scope of the project
could be changed to focus primarily on the parser if there is concern over the
size of the project.</p>
<ol>
<li><strong><a name="iperl_executable" class="item"><strong>iperl</strong> executable</a></strong>
<li><strong><a name="iperl_manual_page" class="item"><strong>iperl</strong> manual page</a></strong>
<li><strong>IPerl::Parser</strong>
<li><strong>IPerl::Introspection</strong>
</ol>
<p>
</p>
<h2>Project Details</h2>
<p>Currently IPerl is a single perl executable (<strong>iperl</strong>). I plan to write a manual
page, package it, and put it on CPAN. In addition, I would build and
incorporate a specialized Perl parser (IPerl::Parser). I would also like to
move some of its functionality (dealing with introspection on the symbol table)
into IPerl::Introspection.</p>
<p>There are also some limitations in <strong>iperl</strong> that I would like to fix. The two
most obvious are the broken <code>my()</code> function (I plan to rewrite top-level
<code>my()</code> invocations into <code>our()</code>) and the requirement that subsequent lines of
continued statements/blocks begin with whitespace (which is not very Perly and
gets dangerously close to Python).</p>
<p>I have already written a fairly good Perl parser (incorporated into the
perl-mode of my text editor) which I should be able to port into <strong>iperl</strong> (as
IPerl::Parser) to accomplish both of these. The parser does not generate a
parse-tree as such, but does tokenize perl input, respecting Perl's rules
regarding <code>m//</code>, <code>s///</code>, heredocs, etc.</p>
<p>
</p>
<h2>Inch-stones</h2>
<p>Here are the steps of the project at a relatively fine granularity:</p>
<ol>
<li><strong>port Perl grammar(s) into Perl</strong>
<p>This involves porting the grammars I have defined in Python into Perl, and
suitably replicating their behavior. Since I am not worried about any
highlighting, generating parse trees, or updating changes in the source
document it should be relatively easy (since much of the python module's
functionality concerns those features).</p>
</li>
<li><strong>build IPerl::Parser</strong>
<p>Once I have Perl grammars, I need to build the IPerl::Parser package and write
POD for it. I expect the package to allow you to instantiate a parser, feed it
input, and read a stream of tokens. It will also have hooks allowing subclasses
to install filters or callbacks (which is what I expect the <strong>iperl</strong> program
will do to translate <code>my()</code> into <code>our()</code>).</p>
</li>
<li><strong>test IPerl::Parser</strong>
<p>At this point I will need to run test cases to determine if the parser is
working well. I have access to about 300,000 lines of object-oriented Perl
(through my work) which I can use. I will also write some simple tests of
various corner cases that I can imagine running into. The two features that I
need to work well are detection of <code>my()</code> and block detection. Since I can
already handle the second in my editor, I do not expect there to be any
insurmountable problems.</p>
</li>
<li><strong><a name="incorporate_iperl_parser_into_iperl" class="item">incorporate IPerl::Parser into <strong>iperl</strong></a></strong>
<p>This involves subclassing IPerl::Parser to translate <code>my()</code>, and also to
determine whether a block remains open at the end of the input. Currently
<strong>iperl</strong> detects opened blocks by matching input against <code>m/{ *$/</code>, which is
less robust.</p>
</li>
<li><strong><a name="test_iperl" class="item">test <strong>iperl</strong></a></strong>
<p>This step is similar to the <strong>test IPerl::Parser</strong> step, but for the <strong>iperl</strong>
program itself.</p>
</li>
<li><strong><a name="write_manual_page_for_iperl" class="item">write manual page for <strong>iperl</strong></a></strong>
<p>This involves writing a manual page which explains how to invoke <strong>iperl</strong>, how to
use it effectively, and documenting its output, known bugs and other
strangeness. It will probably also attempt to give an overview as to how its
internals work.</p>
</li>
<li><strong>abstract introspection code into IPerl::Introspection (optional)</strong>
<p>If time permits, it would be nice to abstract out the code for inspecting the
symbol table so that other modules could benefit from it. This would be useful
for packages that want relatively high-level debugging and for perl programs
that would like to embed an interactive interpreter (a trick that python
programs have been able to do for some time).</p>
</li>
</ol>
<p>
</p>
<h2>Project Schedule</h2>
<p>The project would require about 50 hours of work. This would probably be split
into 12-13 4-hour increments (with a couple of 8 hours days in there). I expect
to be able to work about 2 days a week on the project. By this estimate
(building in time for unforeseen delays) it should take no more than 12 weeks.
I am able to start on it as soon as the project is approved.</p>
<p>
</p>
<h2>Bio</h2>
<p>I am currently the Lead Systems Programmer for Solutions for Progress, Inc. The
Systems team maintains and develops the Benefit Bank [1], a mod_perl
application developed in object-oriented Perl. While teaching non-programmers
at work how to write Perl, I developed <strong>iperl</strong> [2] as a pedagogical tool, and
eventually released it.</p>
<p>I have authored several pieces of open-source software, including <strong>pmacs</strong> [3]
(an Emacs-like editor which is better at highlighting and editing Perl) and
<strong>mpy3</strong> (an extensible curses-based music player) [4].</p>
<dl>
<dt><strong><a name="1_https_selfserve_thebenefitbank_com" class="item">[1] <a href="https://selfserve.thebenefitbank.com">https://selfserve.thebenefitbank.com</a></a></strong>
<dt><strong><a name="2_http_www_bearhome_net_erik_iperl" class="item">[2] <a href="http://www.bearhome.net/erik/iperl">http://www.bearhome.net/erik/iperl</a></a></strong>
<dt><strong><a name="3_http_www_bearhome_net_erik_pmacs" class="item">[3] <a href="http://www.bearhome.net/erik/pmacs">http://www.bearhome.net/erik/pmacs</a></a></strong>
<dt><strong><a name="4_http_www_bearhome_net_mpy3" class="item">[4] <a href="http://www.bearhome.net/mpy3">http://www.bearhome.net/mpy3</a></a></strong>
</dl>
]]>
Comments (14)
This grant seems like it would reproduce some large portions of PPI and Devel::REPL. Also, all this in 50 hours for $1,000? That seems grossly unrealistic.
What benefits would this offer over Devel::REPL with the Devel::REPL::Plugin::MultiLine::PPI plugin?
fyi: i use zoidberg for my interactive perl needs.
Hi all! I was checking the wrong site to see when this would get posted, so I only just saw these comments. I'm at work but will respond as soon as I get a chance.
I must admit that I didn't know about PPI, and it seems clearly superior to writing my own parser. So please consider that part of the proposal changed to "use PPI to implement long lines" which now looks pretty trivial.
I had already seen Zoidberg, Psh, and a couple of other homebrew interactive interpreters. Zoidberg seems more focused on being a login shell: for instance, it doesn't return the result of evaluation by default, which means that one is constantly typing "print" on every other line (especially for one-liners). It also lacks tab completion on variables or functions.
REPL is nice. I just noticed Lexical::Persistence (a feature that I dearly wanted but couldn't find anywhere), so that's good. Unfortunately I don't find the "re.pl" script to have great defaults (for instance, it doesn't turn on any tab completion by default).
Furthermore, it seems like re.pl uses __PACKAGE__ in a strange way. That is, the input "sub abc {}" defines a function called 'abc' in the package 'Devel::REPL', but the input "abc()" tries to call 'abc' in the package 'main'. This ends up being really annoying and kind of defeats the purpose (and certainly does not comply with how perl would interpret the code).
That said, it would be possible to patch REPL to handle my issues, and it already has a reasonable feature set. At this point I am going to email the maintainer to get his opinion on this stuff.
For the purposes of this proposal, please consider the following changes:
1. use PPI to detect multi-line constructs
2. use Lexical::Persistence to allow my() to work (rather than the mapping strategy I talked about).
Those should substantially reduce the amount of work required. My apologies for failing to find them.
Erik, at this point I think the best thing to do is talk to mst (Devel::REPL maintainer), figure out what you'd like to do with Devel::REPL, and submit a new proposal for the next grant period.
Typically, a proposal that proposes to work on someone else's module will also be vetted by that other person, because if they don't want you to do the work, it's not worth approving the grant.
I'm the Devel::REPL author and you can find the team on #devel-repl on irc.perl.org
I've you don't IRC, please contact me at mst (at) shadowcat.co.uk and I'll sort you out with a commit bit and a branch to play with.
Also, your comment about __PACKAGE__ is either a misunderstanding or a bug you've failed to report. It doesn't happen for any of the users who actually talk to me.
The defaults are conservative and the --profile option allows you to pick a whole different basic setup; I think the current default one is probably "what I like" and would be happy to give consideration to a more newbie-friendly set of defaults since the feature to override them is already there.
Looking forward to speaking to you.
I don't think you are quite correct about the Devel::REPL model. If you want, ping me on #devel-repl (on irc.perl.org), and I will help you get your proposal working. I don't think it will take more than an hour, actually. Devel::REPL is very flexible once you are comfortable with the model.
Anyway, the benefit is that the Perl community gets your thing, and we still have $1000 to spend on something else!
At this point it seems clear that I should withdraw the proposal request and concentrate on talking to the Devel::REPL community. I've emailed a more detailed breakdown of my thoughts to Matt and will be heading to #devel-repl.
Thanks for all the feedback and suggestions.
Hi. just a last note saying he's here (i.e. on the channel), we've found him and it looks like we're going to have fun working together.
This should be fun! :D
Perl has an internal interpreter "eval", all perl scripts are evalled first for handle BEGIN, END blocks, it is not necesary aa extern interpreter.
You misunderstand the intent of the grant. It is to provide an /interactive/ interpreter to allow users to test things /interactively/. This is invaluable for noodling around and figuring out problematic bits of code, much like a debugger. In fact, the Perl 5 debugger has been used as a repl (another name for this kind of interpreter) for a long time, but it's lousy at it.
Many, many languages have this kind of program, and it's very useful. It is not really related at all to eval{}, BEGIN, or END.
Maybe you are right, but I need an example of how this iperl works, I dont see how it will work, and perl doesn't seems to many languages, it's unique.
The Perl Debugger, chapter 20, programming perl 3rd edition.
I´ve started learning to program with Python. The IDLE Interpreter is a very helpfully tool especially for absolutly beginners. Now I am forced to learn Perl and I really miss some Application like IDLE.
So good luck for the Project, hope i can test it very soon.
