Hague Grant Application: Implementation of Macros in Rakudo

Thu, 08-Sep-2011 by Karen Pauley  edit post

We have received the following Hague Grant application from Carl Mäsak. Before we vote on this proposal we would like to have a period of community consultation. Please leave feedback in the comments or if you prefer send email with your comments to karen at perlfoundation.org. ## Name Carl Mäsak ## Project Title Implementation of macros in Rakudo ## Synopsis This project will provide Rakudo Perl with AST-based macros. Recently I started discussing various approaches to implementation with Patrick Michaud, Jonathan Worthington, and Larry Wall. The design we arrived at points to a concrete sequence of steps to implement macros in Rakudo. ## Benefits to Perl 6 Development Macros are a Perl 6.0.0 feature that has been discussed since A06 was released back in 2003. The 6.28318 milestone of Pugs promises to deliver macros, but Pugs was released only up to version 6.2.13 before the project went on indefinite hiatus. Neither Rakudo nor Niecza implement macros yet; the closest thing to a Perl 6 macro implementation is a few unpublished experimental prototypes by Stephen Weeks over the years. Furthermore, a number of online documents mention macros and their projected semantics; viz. A06, E06, S06, and lately some examples on Rosetta Code. The descriptions of macros and their capabilities in these documents are often unclear, confused, contradictory, or just plain wrong. This is the typical "liquid" phase of specifications, where no implementation exists to validate or falsify the claims made in the descriptions of a feature, and so explanations and expectations vary all over the map. There's a handful of tests for macros in the spectest repository already, testing such things as macro operators and hygienic macros. More tests, and a more systematic enumeration of test cases, would be needed in order for macros to have sufficient test coverage. This project aims to provide a powerful, extensive, robust first implementation of macros with good test coverage for Rakudo Perl. ## Deliverables D1. Macro declarations and invocation. Rudimentary quasiquotes. D2. Quasiquote splicing. D3. Hygienic macros and the `COMPILING::` pseudopackage. D4. "Delayed" declarations of routines and types within quasiquotes. D5. Updates to the S06 to align the spec with the implementation. D6. Spectests to cover the implemented features. D7. Regular reports on implementation progress. ## Project Details This project will bring the solidity of the specification from "liquid" to "sludgy"; that is, a first implementation of macros will be created, from which the specification can be improved, conclusions can be drawn about what we *really* want from macros, and further course corrections can be made to specification and implementation(s) so that the the status of the feature can eventually, at a point outside of the scope of this project, be considered "solid". The D1..D4 deliverables are discrete steps along the way to a full implementation of macros. The D1 deliverable implements the simplest kind of macro that could possibly work. This partial implementation will allow macros to be declared and then called from code. There are three ways to invoke a macro: as a listop, as a function, and as an operator. All these ways will be provided by this deliverable. With splicing in D2, quasiquotes go from being "rudimentary" to being fully-implemented. The "splicing" part means injecting Perl6::AST objects into the quasiquote, in analogy with how variables inject string values into double-quoted strings. This way, macros will be able to create different kinds of code depending on their parameters. Note that a quasiquote is parametric at macro parse-time but always fully qualified after it has been evaluated as the macro is executed. By design, quasiquotes are always declared as being contained in a block of their own -- however, the block is inlined as the macro is applied. "Hygienic macros", delivered through D3, means that a variable `$x` in the quasiquote won't collide with a variable `$x` declared in a lexical scope in which its macro was invoked. (This will likely need to be done with `gensym` rewriting of the quasiquote's variable.) The `COMPILING::` pseudopackage is an explicit mechanism to conflate the two variable names in an orderly fashion. It will also be provided by this deliverable. The big deliverable is D4, because no mechanism like that exists in Rakudo today. As the Perl 6 grammar in Rakudo parses declarations, it triggers actions to register type names and routine names in various scopes. A quasiquote is different, because at the point it is parsed, the declarations are not supposed to result in names being registered in the normal locations. (Name registration still needs to happen, but only at the point of macro application.) For D4, a mechanism needs to be created which (1) registers the name in the quasiquote somehow, so that declared names are recognized inside the quasiquote, and (2) stores them as "registration events" for when the quasiquote is incorporated into mainline code. Note that, since quasiquotes are still parametric at macro parse-time, the registration events need to be parametric in much the same way. For D5, some required specification changes are already known, but doubtless the need for making changes will also come up as an implementation emerges. ## Project Schedule The project will take approximately four calendar months. The work with the specification (D5), tests (D6), and project reports (D7) are ongoing throughout the project; other work, assuming the grant is accepted in late September, is scheduled as follows: Early October: D1 complete. Late October: D2 complete -- halfway point. Late November: D3 complete. Mid-January: D4 complete. By mid-January, D5, D6, and D7 will also be "complete" in the sense that reports will have been regularly made according to the schedule, tests will have been written along with or in anticipation of implementing features, and (the non-conjectural parts of) the spec will, to the best of my efforts, align with and describe the provided macro implementation. Similarly, at the halfway point, D5/D6/D7 will be caught up with D1/D2. ## Report Schedule I will be able to allocate about 16 hours per week for this project. I plan to blog regularly on [http://strangelyconsistent.org](http://strangelyconsistent.org). By "regularly", I mean at least once every three weeks. ## Public Repository Work will take place in a branch in Rakudo's github repository. The change logs will be viewable through github's web interface. ## Grant Deliverables ownership/copyright and License Information All copyrights for this work, just as for Rakudo itself, will be held by The Perl Foundation, and licensed under the Artistic License 2.0. ## Bio I have been an active Perl 6 community member since 2005, and an active bug reporter and occasional committer to Rakudo Perl since 2008. I've wanted to use macros in Perl 6 since A06 came out. This project will require familiarity with grammars/actions, the meta-object protocol, and serialization contexts. In previous years, I've worked a lot with grammars, and even ported an early Perl 6 grammar engine from PIR to Perl 6. I have a rough grasp of Jonathan Worthington's work on the MOP and serialization contexts, and I'm confident that I'll get up to speed about the rest. I have a firm grasp of Perl 6 semantics, having spent the past three years writing mid-sized Perl 6 applications. I know a thing or two about the limits and the internals of Rakudo, having submitted more than 1100 RT bug reports on it since 2008. Perhaps more importantly, these two activities give me a sense of my own capabilities with respect to implementing a moderately advanced feature in mostly-Perl 6 inside Rakudo. ## Amount Requested 4 months at 40% speed and US$5,000/full month == US$8,000 I'd suggest US$4,000 paid on successful completion up to the halfway point, and US$4,000 paid on successful completion of the whole grant.

---

Comments (9)

---

[ 931 ] | Fri, 09-Sep-2011 by perlpilot

I have long interacted with masak on #perl6 and believe him to be passionate and capable. Plus, finally (!) having an implementation of macros would be a boon to the Perl 6 community and would pave the way for implementations for the other Perl 6 compilers.

I fully support the award of this grant to masak.

---

[ 932 ] | Fri, 09-Sep-2011 by tadeuszsoå_nierz

I cannot wait already. Lots of compiler implementation already seems like it should have been done with macros. It's the missing piece and one of the key and killer feature of Perl 6 itself.

As for Carl's capabilities, I don't think he needs any comments to be considered a fitting candidate for this task. I'm both hands up for this grant.

---

[ 933 ] | Sat, 10-Sep-2011 by perlgeek_de

Carl is a very valuable contributor to many Perl 6 projects, and I think he is well capable of both discovering mostly unknown Perl 6 terrain, and implementing it.

So please accept this grant proposal.

---

[ 934 ] | Sat, 10-Sep-2011 by martinberends

I think deliverables 1, 2, 3, 5, 6, 7 are low-risk as a first half and that 4 is high-risk and should perhaps be done in a separate project. I also believe macros should be used within Rakudo itself to improve parts of the implementation, and that this deliverable or goal should be added as further justification to the proposal.

---

[ 935 ] | Sat, 10-Sep-2011 by sorear

Carl Mäsak is the right person for this; he has an excellent track record when it comes to pushing the boundaries of the Perl 6 specification and he has significant Perl 6 implementation experience.

My main concern with this proposal is that D4 could prove to be effectively impossible within the timeframe. I think there should be flexibility to drop it if this proves to be the case; even without D4 macros would still be very useful.

No comment on the funding rate.

---

[ 936 ] | Sat, 10-Sep-2011 by jimmyz

I'm +1 to implementation of macros in Rakudo, it's a great and important feature in Perl 6. But, I'm not sure whether the amount request is correct or not, I'd like to see a reappraisal of amount request.

---

[ 937 ] | Sun, 11-Sep-2011 by jonathanworthington

Macros are one of those under-explored areas of the Perl 6 specification that really need an implementation; all the parts of the spec we consider stable today are ones backed by implementations. Therefore, for the Perl 6 project as a whole, the work done under this grant would be highly valuable. Of course, for Rakudo it's also of great value: we get an implementation of macros, and have another very capable developer getting a deeper understanding of more areas of Rakudo's internals.

As Carl said in the proposal, and various others who have commented have pointed out, D4 is the most challenging part of the grant. If this had been proposed a year ago, I'd have been concerned about whether anybody could pull it off; with the architecture we currently have in Rakudo, I believe the problem to be vastly more tractable. That certainly doesn't make it easy to get right, but from discussions I've had with Carl recently, I'm comfortable that he's approaching the problem in the right kind of way. I'd rate D4 as tricky, but - with a little thought and patience - decidedly achievable.

So, overall I'd be delighted to see this work happen, and I'm comfortable Carl can deliver. So, the grant gets my vote.

---

[ 938 ] | Wed, 21-Sep-2011 by solomonfoster

It's a very worthy goal, and Mäsak is an excellent choice to get it done. I'm highly in favor.

---

[ 939 ] | Wed, 19-Oct-2011 by carlmã_sak

Just wanted to leave some more detailed information about D4 and other related bits.

The tl;dr of it all is that things are increasingly coming into focus, and D4 is more pervasive and less extricable, but seemingly also more tractable, than I knew when I wrote the grant application.

---