Grant Proposal: RakuAST
Sat, 18-Apr-2020 by
Coke
edit post
<,
[video](http://jnthn.net/papers/2020-gpw-realizing-raku-macros.mp4)), an
abstract syntax tree for the Raku language that will become part of the
language specification. I have also released my work on it so far, which is in
a [branch](https://github.com/rakudo/rakudo/tree/rakuast) in the Rakudo
repository.
So far, I've worked on it during spare moments. The purpose of this grant is
to accelerate progress by enabling me to spend more of my working time on it.
While I already am funded through an ongoing Performance and Reliability
grant - which I shall continue to work on - the RakuAST work does not naturally
fit within its scope, thus this grant application.
# Your name
Jonathan Worthington
# A discussion on the benefits of the project to the Perl Community
An AST can be thought of as a document object model for a programming language.
The goal of RakuAST is to provide an AST that is part of the Raku language
specification, and thus can be relied upon by the language user. Such an AST
is a prerequisite for a useful implementation of macros that actually solve
practical problems, but also offers further powerful opportunities for the
module developer. For example:
* Modules that use Raku as a translation target (for example, `ECMA262Regex`,
a dependency of `JSON::Schema`) can produce a tree representation to `EVAL`
rather than a string. This is more efficient, more secure, and more robust.
(In the standard library, this could also be used to realize a more
efficient `sprintf` implementation.)
* A web framework such as `Cro` could obtain program elements involved in
validation, and translate a typical subset of them into JavaScript (or
patterns for the HTML5 `pattern` attribute) to provide client side
validation automatically.
RakuAST will also become the initial internal representation of Raku programs
used by Rakudo itself. That in turn gives an opportunity to improve the
compiler. The frontend compiler architecture of Rakudo has changed little in
the last 10 years. Naturally, those working on it have learned a few things in
that time, and implementing RakuAST provides a chance to fold those learnings
into the compiler. Better static optimization, use of parallel processing in
the compiler, and improvements to memory and time efficiency are all quite
reasonable expectations.
However, before many of those benefits can be realized, the work of designing
and implementing RakuAST, such that the object model covers the entire semantic
and declarational space of the language, must take place. This grant focuses
on that work.
# A list of deliverables
1. Class and role implementations defining an document object model for the
Raku language, including both the regex and string sub-languages. These
should all be constructable and introspectable from within the Raku
language.
2. Generation of QAST, the backend-independent intermediate representation,
from RakuAST nodes, such that one can execute an AST. This functionality is
exposed as an overload of `EVAL`, since it's ultimately the same operation;
the RakuAST form of `EVAL` can skip the parsing stage entirely, since it
already has the AST.
3. Tests that cover `EVAL` of RakuAST. Every node should be test covered.
4. Integration of RakuAST into the compilation process. This functionality
can be behind an environment variable rather being the default mode of
compilation; this is to avoid a situation where there is pressure to
enable it by default in order to consider the grant complete, when it
would serve Raku users better to make the switch when it is really
ready for that. However, it should be good enough to pass at least
90% of the test suite on the MoarVM backend.
# Project details and a proposed schedule
In *May* I would continue work on the design and implementation of the AST
to cover more of the Raku language. I'll continue to work test-first and
implement compilation of nodes in order they can be `EVAL`'d and produce the
required meta-objects. The goal is to cover a broad range of the main
language.
In *June* I would start work on a means to get the compiler frontend to use
RakuAST, most probably enabled by an environment variable. This need not deal
with such concerns as `BEGIN`, module usage, or precompilation, but it should
be possible to run various simple scripts in this mode successfully.
In *July* I would take on modeling the regex and string sub-languages in
RakuAST, fill out most remaining gaps in the main language, and integrate
these into the compiler frontend as I go.
In *August*, I would focus on some of the more advanced integration with the
compiler frontend, including module usage, `BEGIN`, `constant`, and making
sure any problems with the interaction of RakuAST and precompilation are taken
care of. This should open the way to passing a significant portion of the
specification test suite in RakuAST mode.
# Your biography
I am the founder and architect of MoarVM, the virtual machine used by the
majority of Raku users, and architect of the Rakudo compiler. I led the
previous significant overhaul of the Rakudo compiler, which took place
around a decade ago. I work at Edument, where my roles include product
lead for the Comma IDE for Raku. My experience designing the program
representation there is highly relevant for the work on RakuAST. I hold a
degree in computer science from the University of Cambridge, and aside
from my work on Raku have participated in a range of commercial projects
in the compiler and developer tooling space, including in an architectural
capacity.
# The requested amount for the Project Grant, preferably in US dollars.
$8,500 USD
]]>
Comments (6)
Yes please, this sounds great!
yes please! sounds like a very sensible approach to the problem
I'm highly in favor of funding this grant.
Jonathan has demonstrated countless times that he can deliver high-quality software in the Rakudo compiler and underlying VM, and that his design ideas tend to work out very well.
I also think that a higher-level AST as we have now both helps in terms of performance (because it offers more data for optimization passes to work with) and with advancing macros.
The arguments for ReakuAST both in the grant application and in the video/slides provided are very convincing. Who else if not Jonathan will be able to provide a well designed and implemented solution. The requested funds sound like a bargain.
This feels like a natural next step and a great way of smoothing out the waterbed of complexity between the various layers and internal representations within the Raku 'stack'.
jnthn++ has contributed at all levels of the Raku architecture and he is uniquely qualified to balance all the design dimensions needed to successfully deliver this.
I'm excited about macro support. I'm also keen to see how the changes can help Raku's speed and memory performance. This work will lay the foundation for these improvements.
+1
Aleks-Daniel Jakimenko-Aleksejev wanted to comment, but was unable to post directly: his comment appears below.
--
Everything about this grant proposal sounds extremely uplifting. I can already see potential uses for RakuAST from language user's perspective (after all, nowadays I find myself spending more time using Raku than working on Raku itself). If RakuAST gets us closer to working macros then this work is truly invaluable. For some context, Rakudo Star announcements until 2019.03 used to say that “advanced macros” are not quite there yet and will appear in upcoming releases. This was indeed one of the big promises a lot of people were looking forward to. However, the work on macros was so stagnant that in the most recent announcements this mention was removed altogether. I'm happy to know that we will likely be able to get back on track by using RakuAST as one of the building blocks. As for the potentially improved and silghtly revamped Rakudo architecture, that would be great, although I'm a bit worried that this work will add another layer. That said, nowadays we have a lot of opportunities for noticing performance/memory regressions, so I'm sure it'll be possible to hold this work to high standards. And above all I'm extremely happy that it's jnthn out of all who is willing to make this all happen, as I have no doubt that they will be able to get RakuAST project to completion and in the best way possible. I hope this grant proposal is accepted.
