Grant Application: RakuAST
Fri, 26-Apr-2024 by
Saif Ahmed
edit post
<.
A member of the Raku Steering Council, Stefan is also an author of several Perl 5 modules including Inline::Python and (of course) Inline::Perl6. This Grant is to help advance AST or Abstract Syntax Tree. This is integral to Raku internals and allows designing and implementation of new language components, that can be converted into bytecode for execution by the interpreteter or "virtual machine" more easily that trying to rewrite the interpretter. Here is an [excellent intro](https://dev.to/lizmat/rakuast-for-early-adopters-576n) by [Elizabeth Mattijsen](https://liz.nl/)
<hr>
### Project Title: Taking RakuAST over the finish line
<img src="_ra-rel-mnt_/simplecas/fetch_content/6f840394561d9857b5504b2cb1df0300d1e2f848/RakuAST.png" style="max-width:50%;" />
#### Synopsis
There is a grant called RakuAST granted to Johnathan Worthington that is still listed as running. Sadly Johnathan has moved on and is no longer actively developing the Rakudo core. However the goal of his grant is still worthy as it is one of the strategic initiatives providing numerous benefits to the language. I have in fact already taken over his work on RakuAST and over the last two years have pushed some 450+ commits which led to hundreds of spectests to pass. This work was done in my spare time which was possible because I had a good and reliable source of income and could at times sneak in some Raku work into my dayjob. I can no longer claim that Raku is in any way connected to my day job and time invested in Raku comes directly out of the pool that should ensure my financial future. In other words, there's a real cost for me and I'd like to ask for this to be offset by way of a grant.
#### Benefits to Raku
This is mostly directly taken from the RakuAST grant proposal as the goal stays the same:
An AST can be thought of as a document object model for a programming language. The goal of RakuAST is to provide an AST that is part of the Raku language specification, and thus can be relied upon by the language user. Such an AST is a prerequisite for a useful implementation of macros that actually solve practical problems, but also offers further powerful opportunities for the module developer. For example:
* Modules that use Raku as a translation target (for example, ECMA262Regex, a dependency of JSON::Schema) can produce a tree representation to EVAL rather than a string. This is more efficient, more secure, and more robust. (In the standard library, this could also be used to realize a more efficient sprintf implementation.)
* A web framework such as Cro could obtain program elements involved in validation, and translate a typical subset of them into JavaScript (or patterns for the HTML5 pattern attribute) to provide client side validation automatically.
RakuAST will also become the initial internal representation of Raku programs used by Rakudo itself. That in turn gives an opportunity to improve the compiler. The frontend compiler architecture of Rakudo has changed little in the last 10 years. Naturally, those working on it have learned a few things in that time, and implementing RakuAST provides a chance to fold those learnings into the compiler. Better static optimization, use of parallel processing in the compiler, and improvements to memory and time efficiency are all quite reasonable expectations. We have already seen that the better internal structure fixes a few long standing bugs incidentally. However, before many of those benefits can be realized, the work of designing and implementing RakuAST, such that the object model covers the entire semantic and declarational space of the language, must take place. This grant focuses on that work.
#### Project Details
1. Based on previous development velocity I expect do do some 200 more commits before the RakuAST based compiler frontend passes both Rakudo's test and the Raku spectest suites.
2. Once the test suites pass, there will be some additional work needed to compile Rakudo itself with the RakuAST-frontend. This work will center around bootstrapping issues.
Considering the amount of work these items already will be, I would specifically exclude work targeted at synthetic AST generation, designs for new macros based on this AST, and anything else that is not strictly necessary to reach the goal of the RakuAST compiler frontend becoming the default.
#### Schedule
For the test and spectest suites I would continue my tried and proven model of picking the next failing test file and making fixes until it passes. Based on current velocity this will take around 6 months. However there's hope that some community members will return from their side projects and chime in.
#### Amount Requested
$10,000 for an estimated 200 hours of work.
#### Bio
I have been involved in Rakudo development since 2014 when I started development of Inline::Perl5 which brings full two-way interoperability between Raku and Perl. Since then I have helped with every major effort in Rakudo core development like the Great List Refactor, the new dispatch mechanism and full support for unsigned native integers. I have fixed hundreds of bugs in MoarVM including garbage collection issues, race conditions and bugs in the specializer. I have made NativeCall several orders of magnitude faster by writing a special dispatcher and support for JIT compiling native calls. I replaced a slow and memory hungry MAST step in the compilation process by writing bytecode directly, have written most of Rakudo's module loading and repository management code and in general have done everything I could to make Rakudo production worthy. I have also been a member of the Raku Steering Council since its inception.
#### Supporters
Elizabeth Mattijsen, Geoffrey Broadwell, Nick Logan, Richard Hainsworth
]]>
Comments (9)
As a member of the raku community, I am definitely excited about this work, and confident in Stefan's ability to deliver on it.
100% support from me.
+1
While I'm no longer in a place to contribute much to Raku, I still very much believe RakuAST being completed is of high value for the language's continued evolution and can be a key enabler of better tooling. Stefan has the required experience and stubbornness to see it through to completion - and I'd be very glad to see that happen.
Stefan's work on RakuAST demonstrates quite capably that he is very well-positioned to bring this work across the finish line and thus unlocking a new horizon of potential for Rakudo internals and tooling.
I strongly endorse this application.
I have authored 19 Raku modules to date. I have used Stefan's work on Inline::Perl and Inline::Python and these are solid contributions to the Raku codebase.
I heartily endorse this application.
+1
+1 from me
Stefan has, by his past contributions to the RakuAST effort, proven, that he is the right person to see this project through. As I like to see the effort come to completion, I fully support this grant.
