Grant Proposal: Future::AsyncAwait

9 Comments

The Grants Committee has received the following grant proposal for the Sep/Oct round. Before the Committee members vote, we would like to solicit feedback from the Perl community on the proposal.

Update: You have until October 17th! Review the proposal below and please comment here by October 10th, 2018. The Committee members will start the voting process following that.

Future::AsyncAwait

  • Name:

    Paul Evans

  • Amount Requested:

    GBP 4,800 (GBP 200/day for 24 days)

    At time of writing, equivalent to USD 6,256 or EUR 5,388.

Synopsis

Fix and polish the implementation of the Future::AsyncAwait CPAN module by resolving the bugs that currently prevent it from being useful.

Benefits to the Perl Community

This CPAN module implements the async/await syntax, which has emerged as a common standard among various other languages. At the time of writing this proposal, C#, Python 3, ECMAScript 6 (JavaScript), Dart, and Rust all implement this syntax. It centres around the use of "futures" or "promises" as containers of values to be provided by some possibly-asynchronous or background work, and greatly improves on the readability of programs written using them. Logical flow reads very similarly to standard synchronous call-style.

The examples in the CPAN module should help to illustrate the readability advantages, as do other examples written in the other languages mentioned above. There can be little doubt that async/await is gaining traction as the standard syntax for writing control-flow around future-based concurrency across many diverse programming languages.

By using this particular syntax, Perl 5 gains the same semantics with the same syntax as used by these other languages. This helps readers to use the same concepts as they may already be familiar with from using those languages.

Deliverables

The primary deliverable would be a newer version of Future::AsyncAwait whose internal implementation is robust and fully-functional in a variety of test cases and conditions.

A number of known bugs already exist and are collected on RT (at https://rt.cpan.org/Dist/Display.html?Queue=Future-AsyncAwait). Not all of the tickets in this queue are necessary for completion of this project, as some are "wish-list" items or design discussions about possible further extensions or additions.

In addition to this CPAN release, a secondary deliverable will be a set of blog posts and presentations to educate potential users - bringing the new syntax to their attention, and demonstrating its use. Some blog posts may draw contrasts and similarities to other languages, helping to further emphasize the close relationship between the syntax across these languages. In particular, parallels to Python 3 and ECMAScript 6 may be useful, as both of those languages are ones that potential users may also be familiar with.

Project Details

In its current state, the CPAN module implements a perl parser plugin that parses the newly-defined keywords, and provides implementation semantics that work in many simple test cases. The distribution contains a selection of unit test files that demonstrate these cases.

However, the implementation of the suspend/resume functionality behind the keywords is as yet insufficient for many larger, real-world applications to use in existing code. This is due to the incomplete understanding of some of the internal details of the perl interpreter, which causes some situations to end up in a mismatched state, making it crash.

Central to this project's success is performing further research and understanding on these internals. Once the operation of the interpreter is better understood, the shortcomings of the current implementation of Future::AsyncAwait can be identified and improved upon. This may yield further details that need to be understood which have yet to come to light, but with any luck the process will eventually terminate yielding a new syntax extension that is robust and useful.

Inch-stones

Initial progress will be made from a better understanding of certain details of the perl interpreter, primarily on the subject of the interaction between JMPENV_PUSH and PL_top_env. A discussion of the topic can be seen here: https://www.nntp.perl.org/group/perl.perl5.porters/2018/08/msg251864.html.

It is plausible there may be other subjects that arise once this one is fixed, so the project may continue to switch between research and implementation phases, as bugfixes uncover new areas that don't yet work.

There are particular bugs on the RT queue that need fixing; these are:

Additionally to these, to support the secondary goals of developer education, a number of blog posts can be written:

  • A general introduction to the async/await syntax as provided by Future::AsyncAwait. Suitable for all Perl audiences.
  • A comparison between Future::AsyncAwait and similar syntax provided by Python 3. This will serve as a useful comparison between the two languages, and may help to draw some existing Python developers. Where possible it should include some "Perlish rewrites" of some standard Python documentation, to drive home that similarity.
  • A comparison between Future::AsyncAwait and ECMAScript 6, in similar style to above, for similar reasons.

Thirdly, a presentation for a Perl conference can be prepared to demonstrate the new language feature - perhaps as a follow-up to the talk I did for TPCiA in 2017 (https://www.youtube.com/watch?v=Xf7rStpNaT0), except this time to report on some real-world success stories of the feature being used in production cases.

Finally, the details of the perl interpreter discovered along the way can be better documented, so that even if ultimately it proves impossible to fix the implementation of Future::AsyncAwait, at least others can benefit from the new understanding discovered in that research, which may be helpful to other similar projects.

Project Schedule

There are two main phases to this project: research into the existing operation of the perl interpreter, and fixing the existing implementation of suspend/resume semantics as a result of the discoveries made.

The first phase - research - should be achievable with around 10 days of effort through a combination of code review and instrumented builds, resulting in a textual description of the relevant "moving parts" of the interpreter. This includes time to write down documentation of the discoveries.

Based on this understanding, the second phase - fixing the implementation - can take place. This is likely to take a further 10 days including building more unit test files to cover the new test cases.

In addition, the three blog posts and conference presentation will take around 4 days to prepare.

This brings the total to 24 days.

I'm currently a self-employed contractor with approximately three weeks each month taken by existing clients. Working at a rate of 5 days per month, I believe the project will come to a conclusion after around 5 months.

I intend to work at a rate of 5 days per month so regular checkpoints can be established.

Completeness Criteria

At time of writing there are no known CPAN modules able to use Future::AsyncAwait because it is too unstable at present for even the smallest of unit-tests in other code to successfully pass. A useful moment to consider as a completion target may be when the module is sufficiently robust that it can be used as a dependency by at least a few other CPAN modules that currently use Future.

For example, any of my Device::Chip driver modules, that are currently heavily Future-based at present, would greatly benefit using this new syntax.

CPAN shows a great number of other modules using Future by many authors, and any of these could also be used as test-cases for the robustness of Future::AsyncAwait:

https://metacpan.org/requires/distribution/Future?sort=[[2,1]]

Bio

I am Paul Evans, PEVANS on CPAN (https://metacpan.org/author/PEVANS).

I have been a CPAN maintainer for over 10 years, and currently have 155 distributions under my name. Among this set of modules are a number of dual-life core modules - List::Util and IO::Socket::IP being two that may be among the most heavily-depended upon on CPAN. I have a number of XS-based modules, including some such as Syntax::Keyword::Try that provide keyword plugins to extend the Perl syntax. I am familiar with many parts of the core perl interpreter, and am well-known to many of the perl5-porters group.

I maintain a blog on a variety of programming topics, often posting on Perl-related matters. http://leonerds-code.blogspot.com/search/label/perl.

I have spoken at most London Perl Workshops in the past few years, and attend (and sometimes talk at) the European occurance of what was formerly called YAPC::EU, most recently called TPCiG. I maintain a YouTube playlist of recordings of talks I have given. https://www.youtube.com/playlist?list=PL9-uV_AVx5FOzWJIvpuebmyiIuNd4L7GJ

9 Comments

I'd first like to speak to the value of this project, as I see it. As someone that loves Perl but has worked in many polyglot companies, one of the things that I constantly hear is 'how crappy Perl's support for asynchronous tasks' is. This is of course not true, there's several great systems for this on CPAN. However in a way the proliferation is confusing to newcomers to the language and makes people think asynchronous support for Perl is something bolted on, perhaps haphazardly, rather than well integrated into the language. I believe this work will go a long way towards solving that perception problem, as well as being of tremendous value in and of itself. I hope it will become a great point of integration between the various existing systems and move us closer together as a community. I am sure it will make it easier for people working in other languages to come to Perl, since they will already be familiar with the syntax and it will make it easier for those already working in Perl to take advantage of writing better asynchronous code, which is rapidly becoming a 'must have' in the programming world. In short, this is not only something I think Perl must have on CPAN, but also something we should consider for core integration as soon as possible.

Secondly, I want to speak to the ability of Paul Evans to complete this work. I've never worked with Paul on a job, but I've seen him over the years on IRC helping people and slowly but surely building up a fantastic portfolio of work on CPAN. He's been working in and with Futures and asynchronous code for many years. Additionally he has the trust and respect of some of Perl's most well known programers. I feel very confident in his estimations and ability.

In short, this is work that needs to be done and Paul Evans has my confidence! Thanks for taking the time to read my comments; feel free to follow up with me on any points raised should questions arise.

John Napiorkowski (JJNAPIORK)

+1, async/await is an extremely useful construct which is widely used in other languages. I look forward to production-ready version of Future::AsyncAwait and I certainly will use it in my code.

IMHO funding its development would be a great use of TPF's money.

This is awesome work that Paul has done so far - well worth the funding!

This would be a killer feature for Perl5, bringing much-needed parity to the existing async module support.

If it also shines a light on some of the murkier parts of the Perl internals, even better.

I'd also mention Paul's technical posts such as https://leonerds-code.blogspot.com/2018/09/develmat-investigation-into-c-part-3.html as evidence that he can manage a good publicly-available writeup as well as putting the code together in the first place.

Nearly every one of the CPAN distributions I'm personally developing would be simplified by the existence of this module. As such, I'm biased but hugely in favour.

(disclaimer: we are one of the existing clients that Paul mentions, and once this is stable we have every intention of using this module heavily in both our internal and public CPAN code)

+1 to this proposal! Having async/await in perl5 would be awesome!

I strongly support this grant request. Paul does great work and is highly responsive and communicative. This project will greatly benefit the community and Perl as a whole.

Async/Await would be a great addition to the language.
This should get funded.

After finally getting my head around promises in JavaScript it has been a pleasure to be able to use Promises in Perl via https://metacpan.org/pod/Mojo::Promise. Doing the same with async/await will also help me convince other developers at $work that Perl isn't as terrible as the 10+ year old non-strict-single-file-Perl code that they've been hacking on for the last 5 years.

Leave a comment

About TPF

The Perl Foundation - supporting the Perl community since 2000. Find out more at www.perlfoundation.org.

About this Entry

This page contains a single entry by Coke published on October 4, 2018 1:25 AM.

Grant Proposals Sep/Oct 2018 was the previous entry in this blog.

TPF Notes from OSCON 2018 is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

Pages

OpenID accepted here Learn more about OpenID
Powered by Movable Type 6.2.2