2009Q1 Grant Proposal: Interactive Perl Interpreter

| 14 Comments
Name:

Erik Osheim

Email:

[hidden email]

Amount Requested:

$1000

Synopsis

This project aims to build an interactive interpreter for Perl. The goals of this project are to allow pasted Perl code to run line-by-line as it would be in a Perl script, provide tab completion and introspection at least equal to that provided by the Perl debugger, to provide documentation and packaging, and lastly (and optionally) a package to help with introspection.

Benefits to the Perl Community

Interpreted languages like Scheme, Python and Ruby all have interactive interpreters to help people begin programming, learn tricks and eventually delve deeper into the lower-level details of the language. Despite having a robust debugger, Perl has never had precisely such a tool. Several interpreters exist online to try to fill this niche, but none of them address the main problems that the perl debugger faces (multi-line statements and blocks fail, my() does not work as expected).

An interactive interpreter would serve an important pedagogical role. Tutorials are easier to follow with "bash-like" interactive line-editing, history, and tab-completion. "Learn by doing" becomes easier when programming interactively, when errors in syntax or semantics become more immediate.

This project also answers criticisms from other language communities that Perl needs (or at least could use) an interactive interpreter. Even experienced Perl programmers would benefit from being able to test and refine particularly interesting constructions in an interactive shell.

Deliverables

The program and manual page are mandatory deliverables for the project. Depending on feedback from the community and usefulness, the IPerl packages may also be included (possibly with different names). The scope of the project could be changed to focus primarily on the parser if there is concern over the size of the project.

  1. iperl executable
  2. iperl manual page
  3. IPerl::Parser
  4. IPerl::Introspection

Project Details

Currently IPerl is a single perl executable (iperl). I plan to write a manual page, package it, and put it on CPAN. In addition, I would build and incorporate a specialized Perl parser (IPerl::Parser). I would also like to move some of its functionality (dealing with introspection on the symbol table) into IPerl::Introspection.

There are also some limitations in iperl that I would like to fix. The two most obvious are the broken my() function (I plan to rewrite top-level my() invocations into our()) and the requirement that subsequent lines of continued statements/blocks begin with whitespace (which is not very Perly and gets dangerously close to Python).

I have already written a fairly good Perl parser (incorporated into the perl-mode of my text editor) which I should be able to port into iperl (as IPerl::Parser) to accomplish both of these. The parser does not generate a parse-tree as such, but does tokenize perl input, respecting Perl's rules regarding m//, s///, heredocs, etc.

Inch-stones

Here are the steps of the project at a relatively fine granularity:

  1. port Perl grammar(s) into Perl

    This involves porting the grammars I have defined in Python into Perl, and suitably replicating their behavior. Since I am not worried about any highlighting, generating parse trees, or updating changes in the source document it should be relatively easy (since much of the python module's functionality concerns those features).

  2. build IPerl::Parser

    Once I have Perl grammars, I need to build the IPerl::Parser package and write POD for it. I expect the package to allow you to instantiate a parser, feed it input, and read a stream of tokens. It will also have hooks allowing subclasses to install filters or callbacks (which is what I expect the iperl program will do to translate my() into our()).

  3. test IPerl::Parser

    At this point I will need to run test cases to determine if the parser is working well. I have access to about 300,000 lines of object-oriented Perl (through my work) which I can use. I will also write some simple tests of various corner cases that I can imagine running into. The two features that I need to work well are detection of my() and block detection. Since I can already handle the second in my editor, I do not expect there to be any insurmountable problems.

  4. incorporate IPerl::Parser into iperl

    This involves subclassing IPerl::Parser to translate my(), and also to determine whether a block remains open at the end of the input. Currently iperl detects opened blocks by matching input against m/{ *$/, which is less robust.

  5. test iperl

    This step is similar to the test IPerl::Parser step, but for the iperl program itself.

  6. write manual page for iperl

    This involves writing a manual page which explains how to invoke iperl, how to use it effectively, and documenting its output, known bugs and other strangeness. It will probably also attempt to give an overview as to how its internals work.

  7. abstract introspection code into IPerl::Introspection (optional)

    If time permits, it would be nice to abstract out the code for inspecting the symbol table so that other modules could benefit from it. This would be useful for packages that want relatively high-level debugging and for perl programs that would like to embed an interactive interpreter (a trick that python programs have been able to do for some time).

Project Schedule

The project would require about 50 hours of work. This would probably be split into 12-13 4-hour increments (with a couple of 8 hours days in there). I expect to be able to work about 2 days a week on the project. By this estimate (building in time for unforeseen delays) it should take no more than 12 weeks. I am able to start on it as soon as the project is approved.

Bio

I am currently the Lead Systems Programmer for Solutions for Progress, Inc. The Systems team maintains and develops the Benefit Bank [1], a mod_perl application developed in object-oriented Perl. While teaching non-programmers at work how to write Perl, I developed iperl [2] as a pedagogical tool, and eventually released it.

I have authored several pieces of open-source software, including pmacs [3] (an Emacs-like editor which is better at highlighting and editing Perl) and mpy3 (an extensible curses-based music player) [4].

[1] https://selfserve.thebenefitbank.com
[2] http://www.bearhome.net/erik/iperl
[3] http://www.bearhome.net/erik/pmacs
[4] http://www.bearhome.net/mpy3

14 Comments

This grant seems like it would reproduce some large portions of PPI and Devel::REPL. Also, all this in 50 hours for $1,000? That seems grossly unrealistic.

What benefits would this offer over Devel::REPL with the Devel::REPL::Plugin::MultiLine::PPI plugin?

fyi: i use zoidberg for my interactive perl needs.

Hi all! I was checking the wrong site to see when this would get posted, so I only just saw these comments. I'm at work but will respond as soon as I get a chance.

I must admit that I didn't know about PPI, and it seems clearly superior to writing my own parser. So please consider that part of the proposal changed to "use PPI to implement long lines" which now looks pretty trivial.

I had already seen Zoidberg, Psh, and a couple of other homebrew interactive interpreters. Zoidberg seems more focused on being a login shell: for instance, it doesn't return the result of evaluation by default, which means that one is constantly typing "print" on every other line (especially for one-liners). It also lacks tab completion on variables or functions.

REPL is nice. I just noticed Lexical::Persistence (a feature that I dearly wanted but couldn't find anywhere), so that's good. Unfortunately I don't find the "re.pl" script to have great defaults (for instance, it doesn't turn on any tab completion by default).

Furthermore, it seems like re.pl uses __PACKAGE__ in a strange way. That is, the input "sub abc {}" defines a function called 'abc' in the package 'Devel::REPL', but the input "abc()" tries to call 'abc' in the package 'main'. This ends up being really annoying and kind of defeats the purpose (and certainly does not comply with how perl would interpret the code).

That said, it would be possible to patch REPL to handle my issues, and it already has a reasonable feature set. At this point I am going to email the maintainer to get his opinion on this stuff.

For the purposes of this proposal, please consider the following changes:

1. use PPI to detect multi-line constructs
2. use Lexical::Persistence to allow my() to work (rather than the mapping strategy I talked about).

Those should substantially reduce the amount of work required. My apologies for failing to find them.

Erik, at this point I think the best thing to do is talk to mst (Devel::REPL maintainer), figure out what you'd like to do with Devel::REPL, and submit a new proposal for the next grant period.

Typically, a proposal that proposes to work on someone else's module will also be vetted by that other person, because if they don't want you to do the work, it's not worth approving the grant.

I'm the Devel::REPL author and you can find the team on #devel-repl on irc.perl.org

I've you don't IRC, please contact me at mst (at) shadowcat.co.uk and I'll sort you out with a commit bit and a branch to play with.

Also, your comment about __PACKAGE__ is either a misunderstanding or a bug you've failed to report. It doesn't happen for any of the users who actually talk to me.

The defaults are conservative and the --profile option allows you to pick a whole different basic setup; I think the current default one is probably "what I like" and would be happy to give consideration to a more newbie-friendly set of defaults since the feature to override them is already there.

Looking forward to speaking to you.

I don't think you are quite correct about the Devel::REPL model. If you want, ping me on #devel-repl (on irc.perl.org), and I will help you get your proposal working. I don't think it will take more than an hour, actually. Devel::REPL is very flexible once you are comfortable with the model.

Anyway, the benefit is that the Perl community gets your thing, and we still have $1000 to spend on something else!

At this point it seems clear that I should withdraw the proposal request and concentrate on talking to the Devel::REPL community. I've emailed a more detailed breakdown of my thoughts to Matt and will be heading to #devel-repl.

Thanks for all the feedback and suggestions.

Hi. just a last note saying he's here (i.e. on the channel), we've found him and it looks like we're going to have fun working together.

This should be fun! :D

Perl has an internal interpreter "eval", all perl scripts are evalled first for handle BEGIN, END blocks, it is not necesary aa extern interpreter.

You misunderstand the intent of the grant. It is to provide an /interactive/ interpreter to allow users to test things /interactively/. This is invaluable for noodling around and figuring out problematic bits of code, much like a debugger. In fact, the Perl 5 debugger has been used as a repl (another name for this kind of interpreter) for a long time, but it's lousy at it.

Many, many languages have this kind of program, and it's very useful. It is not really related at all to eval{}, BEGIN, or END.

Maybe you are right, but I need an example of how this iperl works, I dont see how it will work, and perl doesn't seems to many languages, it's unique.

The Perl Debugger, chapter 20, programming perl 3rd edition.

I´ve started learning to program with Python. The IDLE Interpreter is a very helpfully tool especially for absolutly beginners. Now I am forced to learn Perl and I really miss some Application like IDLE.
So good luck for the Project, hope i can test it very soon.

Leave a comment

About this Entry

This page contains a single entry by Alberto Simões published on February 6, 2009 5:19 PM.

Find recent content on the main index or look in the archives to find all content.