Grant Proposal: Future::AsyncAwait
Thu, 04-Oct-2018 by
Coke
edit post
< CPAN module by
resolving the bugs that currently prevent it from being useful.
## Benefits to the Perl Community
This CPAN module implements the `async/await` syntax, which has emerged as a
common standard among various other languages. At the time of writing this
proposal, C#, Python 3, ECMAScript 6 (JavaScript), Dart, and Rust all implement
this syntax. It centres around the use of "futures" or "promises" as containers
of values to be provided by some possibly-asynchronous or background work, and
greatly improves on the readability of programs written using them. Logical
flow reads very similarly to standard synchronous call-style.
The examples in the CPAN module should help to illustrate the readability
advantages, as do other examples written in the other languages mentioned
above. There can be little doubt that `async/await` is gaining traction as the
standard syntax for writing control-flow around future-based concurrency across
many diverse programming languages.
By using this particular syntax, Perl 5 gains the same semantics with the same
syntax as used by these other languages. This helps readers to use the same
concepts as they may already be familiar with from using those languages.
## Deliverables
The primary deliverable would be a newer version of [Future::AsyncAwait](https://metacpan.org/pod/Future::AsyncAwait) whose
internal implementation is robust and fully-functional in a variety of test
cases and conditions.
A number of known bugs already exist and are collected on RT (at
[https://rt.cpan.org/Dist/Display.html?Queue=Future-AsyncAwait](https://rt.cpan.org/Dist/Display.html?Queue=Future-AsyncAwait)). Not all of
the tickets in this queue are necessary for completion of this project, as
some are "wish-list" items or design discussions about possible further
extensions or additions.
In addition to this CPAN release, a secondary deliverable will be a set of
blog posts and presentations to educate potential users - bringing the new
syntax to their attention, and demonstrating its use. Some blog posts may draw
contrasts and similarities to other languages, helping to further emphasize the
close relationship between the syntax across these languages. In particular,
parallels to Python 3 and ECMAScript 6 may be useful, as both of those
languages are ones that potential users may also be familiar with.
## Project Details
In its current state, the CPAN module implements a perl parser plugin that
parses the newly-defined keywords, and provides implementation semantics that
work in many simple test cases. The distribution contains a selection of unit
test files that demonstrate these cases.
However, the implementation of the suspend/resume functionality behind the
keywords is as yet insufficient for many larger, real-world applications to use
in existing code. This is due to the incomplete understanding of some of the
internal details of the `perl` interpreter, which causes some situations to
end up in a mismatched state, making it crash.
Central to this project's success is performing further research and
understanding on these internals. Once the operation of the interpreter is
better understood, the shortcomings of the current implementation of
[Future::AsyncAwait](https://metacpan.org/pod/Future::AsyncAwait) can be identified and improved upon. This may yield
further details that need to be understood which have yet to come to light,
but with any luck the process will eventually terminate yielding a new syntax
extension that is robust and useful.
## Inch-stones
Initial progress will be made from a better understanding of certain details of
the `perl` interpreter, primarily on the subject of the interaction between
`JMPENV_PUSH` and `PL_top_env`. A discussion of the topic can be seen here:
[https://www.nntp.perl.org/group/perl.perl5.porters/2018/08/msg251864.html](https://www.nntp.perl.org/group/perl.perl5.porters/2018/08/msg251864.html).
It is plausible there may be other subjects that arise once this one is fixed,
so the project may continue to switch between research and implementation
phases, as bugfixes uncover new areas that don't yet work.
There are particular bugs on the RT queue that need fixing; these are:
- [https://rt.cpan.org/Ticket/Display.html?id=126037](https://rt.cpan.org/Ticket/Display.html?id=126037)
- [https://rt.cpan.org/Ticket/Display.html?id=126036](https://rt.cpan.org/Ticket/Display.html?id=126036)
- [https://rt.cpan.org/Ticket/Display.html?id=125613](https://rt.cpan.org/Ticket/Display.html?id=125613)
- [https://rt.cpan.org/Ticket/Display.html?id=123062](https://rt.cpan.org/Ticket/Display.html?id=123062)
Additionally to these, to support the secondary goals of developer education,
a number of blog posts can be written:
- A general introduction to the `async/await` syntax as provided by
[Future::AsyncAwait](https://metacpan.org/pod/Future::AsyncAwait). Suitable for all Perl audiences.
- A comparison between [Future::AsyncAwait](https://metacpan.org/pod/Future::AsyncAwait) and similar syntax provided by
Python 3. This will serve as a useful comparison between the two languages, and
may help to draw some existing Python developers. Where possible it should
include some "Perlish rewrites" of some standard Python documentation, to
drive home that similarity.
- A comparison between [Future::AsyncAwait](https://metacpan.org/pod/Future::AsyncAwait) and ECMAScript 6, in similar style
to above, for similar reasons.
Thirdly, a presentation for a Perl conference can be prepared to demonstrate
the new language feature - perhaps as a follow-up to the talk I did for TPCiA
in 2017 ([https://www.youtube.com/watch?v=Xf7rStpNaT0](https://www.youtube.com/watch?v=Xf7rStpNaT0)), except this time to
report on some real-world success stories of the feature being used in
production cases.
Finally, the details of the perl interpreter discovered along the way can be
better documented, so that even if ultimately it proves impossible to fix the
implementation of [Future::AsyncAwait](https://metacpan.org/pod/Future::AsyncAwait), at least others can benefit from the
new understanding discovered in that research, which may be helpful to other
similar projects.
## Project Schedule
There are two main phases to this project: research into the existing operation
of the `perl` interpreter, and fixing the existing implementation of
suspend/resume semantics as a result of the discoveries made.
The first phase - research - should be achievable with around 10 days of effort
through a combination of code review and instrumented builds, resulting in a
textual description of the relevant "moving parts" of the interpreter. This
includes time to write down documentation of the discoveries.
Based on this understanding, the second phase - fixing the implementation - can
take place. This is likely to take a further 10 days including building more
unit test files to cover the new test cases.
In addition, the three blog posts and conference presentation will take around
4 days to prepare.
This brings the total to 24 days.
I'm currently a self-employed contractor with approximately three weeks each
month taken by existing clients. Working at a rate of 5 days per month, I
believe the project will come to a conclusion after around 5 months.
I intend to work at a rate of 5 days per month so regular checkpoints can be
established.
## Completeness Criteria
At time of writing there are no known CPAN modules able to use
[Future::AsyncAwait](https://metacpan.org/pod/Future::AsyncAwait) because it is too unstable at present for even the
smallest of unit-tests in other code to successfully pass. A useful moment to
consider as a completion target may be when the module is sufficiently robust
that it can be used as a dependency by at least a few other CPAN modules that
currently use [Future](https://metacpan.org/pod/Future).
For example, any of my [Device::Chip](https://metacpan.org/pod/Device::Chip) driver modules, that are currently
heavily [Future](https://metacpan.org/pod/Future)-based at present, would greatly benefit using this new
syntax.
CPAN shows a great number of other modules using [Future](https://metacpan.org/pod/Future) by many authors,
and any of these could also be used as test-cases for the robustness of
[Future::AsyncAwait](https://metacpan.org/pod/Future::AsyncAwait):
[https://metacpan.org/requires/distribution/Future?sort=\[\[2,1\]\]](https://metacpan.org/requires/distribution/Future?sort=[[2,1]])
## Bio
I am Paul Evans, PEVANS on CPAN ([https://metacpan.org/author/PEVANS](https://metacpan.org/author/PEVANS)).
I have been a CPAN maintainer for over 10 years, and currently have 155
distributions under my name. Among this set of modules are a number of
dual-life core modules - [List::Util](https://metacpan.org/pod/List::Util) and [IO::Socket::IP](https://metacpan.org/pod/IO::Socket::IP) being two that
may be among the most heavily-depended upon on CPAN. I have a number of
XS-based modules, including some such as [Syntax::Keyword::Try](https://metacpan.org/pod/Syntax::Keyword::Try) that provide
keyword plugins to extend the Perl syntax. I am familiar with many parts of the
core perl interpreter, and am well-known to many of the perl5-porters group.
I maintain a blog on a variety of programming topics, often posting on
Perl-related matters. [http://leonerds-code.blogspot.com/search/label/perl](http://leonerds-code.blogspot.com/search/label/perl).
I have spoken at most London Perl Workshops in the past few years, and attend
(and sometimes talk at) the European occurance of what was formerly called
YAPC::EU, most recently called TPCiG. I maintain a YouTube playlist of
recordings of talks I have given.
[https://www.youtube.com/playlist?list=PL9-uV\_AVx5FOzWJIvpuebmyiIuNd4L7GJ](https://www.youtube.com/playlist?list=PL9-uV_AVx5FOzWJIvpuebmyiIuNd4L7GJ)
]]>
Comments (11)
I'd first like to speak to the value of this project, as I see it. As someone that loves Perl but has worked in many polyglot companies, one of the things that I constantly hear is 'how crappy Perl's support for asynchronous tasks' is. This is of course not true, there's several great systems for this on CPAN. However in a way the proliferation is confusing to newcomers to the language and makes people think asynchronous support for Perl is something bolted on, perhaps haphazardly, rather than well integrated into the language. I believe this work will go a long way towards solving that perception problem, as well as being of tremendous value in and of itself. I hope it will become a great point of integration between the various existing systems and move us closer together as a community. I am sure it will make it easier for people working in other languages to come to Perl, since they will already be familiar with the syntax and it will make it easier for those already working in Perl to take advantage of writing better asynchronous code, which is rapidly becoming a 'must have' in the programming world. In short, this is not only something I think Perl must have on CPAN, but also something we should consider for core integration as soon as possible.
Secondly, I want to speak to the ability of Paul Evans to complete this work. I've never worked with Paul on a job, but I've seen him over the years on IRC helping people and slowly but surely building up a fantastic portfolio of work on CPAN. He's been working in and with Futures and asynchronous code for many years. Additionally he has the trust and respect of some of Perl's most well known programers. I feel very confident in his estimations and ability.
In short, this is work that needs to be done and Paul Evans has my confidence! Thanks for taking the time to read my comments; feel free to follow up with me on any points raised should questions arise.
John Napiorkowski (JJNAPIORK)
+1, async/await is an extremely useful construct which is widely used in other languages. I look forward to production-ready version of Future::AsyncAwait and I certainly will use it in my code.
IMHO funding its development would be a great use of TPF's money.
This is awesome work that Paul has done so far - well worth the funding!
This would be a killer feature for Perl5, bringing much-needed parity to the existing async module support.
If it also shines a light on some of the murkier parts of the Perl internals, even better.
I'd also mention Paul's technical posts such as https://leonerds-code.blogspot.com/2018/09/develmat-investigation-into-c-part-3.html as evidence that he can manage a good publicly-available writeup as well as putting the code together in the first place.
Nearly every one of the CPAN distributions I'm personally developing would be simplified by the existence of this module. As such, I'm biased but hugely in favour.
(disclaimer: we are one of the existing clients that Paul mentions, and once this is stable we have every intention of using this module heavily in both our internal and public CPAN code)
+1 to this proposal! Having async/await in perl5 would be awesome!
I strongly support this grant request. Paul does great work and is highly responsive and communicative. This project will greatly benefit the community and Perl as a whole.
Folks, apologies; I think I accidentally deleted 2 real comments when removing the 700+ spam entries that were posted in the past week.
All queued comments have been posted as of this time; if your comment is missing, please re-add it.
Regards.
Async/Await would be a great addition to the language.
This should get funded.
After finally getting my head around promises in JavaScript it has been a pleasure to be able to use Promises in Perl via https://metacpan.org/pod/Mojo::Promise. Doing the same with async/await will also help me convince other developers at $work that Perl isn't as terrible as the 10+ year old non-strict-single-file-Perl code that they've been hacking on for the last 5 years.
First of all, I want to highly endorse this work and Paul's efforts to do it. This is an important project that Perl sorely needs. That said I do want to ask a few minor conditions on the project to support community funding of it.
This mostly revolves around Paul's use of his own Future stack to accomplish the behavior. This is understandable as with many authors we like to use our own stuff. The problem with Future is that it isn't entirely interoperable with other Promise implementations on CPAN. (Note that he and I have discussed this amicably on #mojo). There does exist a standard for Promise implementations across languages called the Promises/A+ spec (https://promisesaplus.com/). Future does not conform with that standard (fair, as it predates it) and due to some non-standard behavior as a result it isn't entirely easy for other implementations to conform to his. According to him, there are several methods that must be implemented in the Future way in order for the proposed AsyncAwait mechanism to work. I would ask that if the community were to support his work (and again, I hope we do) that he be asked to follow the Promises/A+ spec in doing so.
If that cannot be reasonably asked for in the time or support amount requested, then I could ask a lesser goal of having him expose his suspend/resume mechanism, the low level guts, in some higher level way so that other Async/Await mechanisms may be built on top of it. Preferably this would be at the perl level though if it must be in the XS level, I won't object too strongly as long as it is usable.
Once again, I want to reiterate that I want to see this grant supported, I just hope that the result is as broadly applicable as possible.
I would ask that if the community were to support his work (and again, I hope we do) that he be asked to follow the Promises/A+ spec in doing so.
That seems like a reasonable request, but I don't think it'd be as effective as it might first sound.
Firstly, I don't believe this is directly possible in Perl without a better definition of
https://es5.github.io/#x10.3, which is required by
https://promisesaplus.com/#point-34. The footnote brings in a requirement for an "event loop", which is an implementation detail - Future.pm itself does not require this, and a standard "event loop" or "event loop interface" seems a bit out of scope - look at how much time and effort has gone into getting "a MOP into core"!
Given the existence of modules such as Mojo::Promise::Role::Futurify it seems more likely that Promises/A+-compatible implementations can be built on top of Future.pm rather than the other way around? There are several features of Future.pm - for example, support for a distinct "cancelled" state - that are a superset of the Promises/A+ spec. If it's not possible to build the compatibility layer on top of Future.pm, maybe it'd help to have a clear set of requirements that cannot be fulfilled by the current proposal?
This section of the Promises.pm documentation also seems relevant:
We are, with some differences, following the API spec called "Promise/A" (and the clarification that is called "Promise/A+") which was created by the Node.JS community. This is, for the most part, the same API that is implemented in the latest jQuery and in the YUI Deferred plug-in (though some purists argue that they both go it wrong, google it if you care). We differ in some respects to this spec, mostly because Perl idioms and best practices are not the same as Javascript idioms and best practices. However, the one important difference that should be noted is that "Promise/A+" strongly suggests that the callbacks given to then should be run asynchronously (meaning in the next turn of the event loop). We do not do this by default, because doing so would bind us to a given event loop implementation, which we very much want to avoid.
There are a few other Future/Promises implementations in Perl:
- Future::Q - a Perl implementation of http://documentup.com/kriskowal/q/ which is (claimed to be) Promises/A+-compliant
- AnyEvent::Promise - the then method is only documented to support a single callback
- AnyEvent::Promises - callbacks can't return AnyEvent::Promises instances directly, since the chaining behaviour means it'll wait for those to be resolved/rejected
- Evo::Promise::* - mostly builds on other things, but having several different classes here makes it a bit hard to extract a common API
Lastly, the Promises/A+ spec is popular in the JS/ES6 world, but C#'s Task/TaskCompletion is a different beast, as is the shocking mess of C++'s std::future. Python3's equivalent is a superset, but is at least compatible. Scala (and Java) also do their own thing:
https://docs.scala-lang.org/overviews/core/futures.html
(disclaimer: the majority of my CPAN modules have been based on Future.pm for years, so I know that API far better than the other options)
