RakuAST Grant Report from Jonathan Worthington, 2020-09
Tue, 06-Oct-2020 by
Moritz Lenz
edit post
<. His work his highly appreciated, the resting period well-desevered, and we expect his work to continue in a month or a few months.
Here's his progress report, in his own words:
---
In [last month's report](https://news.perlfoundation.org/post/grant-report-jonathan-worthington-rakuast-2020-08), I mentioned
that the RakuAST-based compiler frontend had reached the point of being able to attempt the specification test suite, thanks
to it gaining support for basic `use` statements. For now, it depends on using the standard compiler frontend to precompile the test modules. Back then, it was capable of passing 39 spectest files in full.
Fast forward a month, and that number stands at 209 passing spectest files. On the one hand, that's quite a step forward.
On the other hand, it means there's still quite a bit to go. Part of the challenge here is that spectest files may aim to
test a particular feature, but since a lot of interesting behaviors involve the interactions of features, they often end
up touching quite a lot of the language. Thus implementing support for a particular language construct's compilation via
the new Raku AST does not automatically mean we're now going to be able to pass the spectest file(s) that are most
obviously associated with it. It may, however, mean we unblock some other spectest files that aren't directly about the
construct, but were blocking on it! For example, implementing support for the `.WHAT` type introspection operation resulted in a bunch of new passing spectest files. It's also not just about feature interactions. Tests are code, and people like to factor
code nicely, and that can also entail use of further language constructs beyond those under test.
Here's a summary of the advances made over the last month; generally, when a language construct is mentioned, then RakuAST nodes were developed and also the work was done to construct these in the new RakuAST-based compiler frontend.
* Get multi-part package lookups of types working
* Add the Pod grammar to the new compiler frontend (which won quite a bunch of spectests that contained Pod comments)
* Get single-part package name installation working, so that simple package declarations result in installation of a type name that can be resolved; the key realization here is that installation in a package is most naturally handled as a `BEGIN`-time side-effect
* Get method definitions working, including production of the implicit invocant parameter
* Implement the `self` term and the `.foo` term for calls on the topic
* Implement basic attribute declaration and attribute access
* Implement `.=` (call method and assign) and `.` (call method) infixes; these are interesting in that they parse a postfix after them rather than a term, and this also needed modeling in the AST
* Implement the terms `now`, `time`, `rand`, empty set, `*`, and `**` terms (however not yet the whatever-currying semantics)
* Implement lookups of named constant terms (`True` and `False` being the most obvious examples)
* Implement all forms of colonpair (so far, only as terms rather than as part of a name)
* Wire up compilation of dynamic variables and match variables to the already existing AST nodes
* Implement the literal hash index (`%foo<x>`) and call (`$foo()`) postcircumfixes
* Make `EVAL` work when running the RakuAST-based frontend, so we can use RakuAST to compile a program that calls `EVAL` that uses the RakuAST-based compiler
* Make the new compiler frontend parse and pass along types on lexical variables (the AST already supported this)
* Implement term (`\foo`), optional, named, slurpy, and capture parameters, as well as basic nominal type constraints on parameters
* Implement the hash/block distinction rules for `{...}`
* Properly model empty statements in the AST (they actually need to persist into the AST, because the presence of an empty statement can make a difference in hash/block differentiation)
* Parse `Num` and `Rat` literals, including the angle bracket `Rat` literal form. Also parse other Unicode number forms.
* Implement capture literals
* Support `use lib` in the case that the expression following it is something simple enough to interpret. Also support import tags on `use` statements. (Interpreting these will give us a compilation performance improvement, because we don't have to send the relatively simple expressions all the way through the compilation pipeline and then run the resulting bytecode.)
* Support very simple `BEGIN` expressions (only ones that we can interpret for now)
* Provide `$?FILE` and `$?LINE`. These concepts don't really exist at the AST level, so these are actually constructed by the frontend with the filename or line number from the source.
* Design and implement the foundations for traits. Add RakuAST nodes for the `does`, `hides`, `of` and `returns` traits, and apply such traits to packages and routines (so far, only for traits provided by modules or the setting, rather than those in the current source file).
* Improve the routine definition compilation to enforce return type checks and handle the `return` control exception
RakuAST is a huge effort, and more people working on it, or with it, is going to be essential to
delivering a good end result. Happily, over the last month, `lizmat` has joined in, and with a
bit of input from me has already put RakuAST to use as the target for compiling `printf` formats,
as well as building a deparse feature the renders the AST back into Raku source code. As it grows
more capable, I look forward to see others experimenting with it too, which will provide valuable
feedback - and of course, contributions to the implementation of the AST nodes and their compilation
are also welcomed!
]]>
Comments (0)
