Hague Grant Application

21 Comments

We have received the following Hague Grant application from diakopter.
Before we vote on this proposal we would like to have a period of community consultation for 10 days. Please leave feedback in the comments or if you prefer send email with your comments to karen at perlfoundation.org.

Applicant's Name: diakopter

Project Title

Wrap up Perl 5 Interop for Rakudo Perl 6 on MoarVM (2 months)

Synopsis

Embed Perl 5 in Rakudo's implementation on one of its two newest backends, MoarVM, in order to enable seamless two-way interop between the Perl 6 environment and the Perl 5 environment (9 weeks).

Benefits to Perl 6 Development

Rakudo Perl 6 currently has limited access to Perl 5 modules and CPAN distributions via Blizkost, but only on the Parrot VM. Tobias Leich is building rakudo-p5/v5 (use v5;), which is a reimplementation of Perl 5 in Perl 6, but that effort will be able to support only a small subset of what's on CPAN. (However, see below on how it could help in the future.) Bringing all of CPAN to Rakudo via embedding the Perl 5 interpreter would open up entirely new worlds of opportunities and additional Perl 6 (and Perl 5!) adoption and maintenance paths.

Deliverables & Project Details

We expect Rakudo to be basically done being ported to the JVM by August, and then also to MoarVM not too many weeks after that. I can get much of the coding done before Rakudo is bootstrapped on MoarVM, but nearly all of the writing tests in Perl 6 would occur in the third month. As far as coding goes, I've already sketched out around 40% of the code for the Perl 5 interop.

Long-term, there are many potential Perl 5 Interop deliverables, but to start, it is an extension to MoarVM and Rakudo p6, either distributed separately or optionally built-in (or both), that embeds p5 libperl to enable calling p5 code (interacting with p5 objects) from p6, and vice versa, for the typical use case of using p5 CPAN distributions in p6 code.

Feature details/schedule:

  1. finalize MoarVM's dynamic extension interface protocols & standards (using DynCall - design already pushed to the extops branch). Note: dependency on importing DynCall into MoarVM (from NQP proper). See http://goo.gl/W5mI0 for the design
  2. Skeleton extension "mvmp5" that installs the custom op for p5 interop and installs the custom REPRs into MoarVM/6model (for the p5 object wrappers and others)
  3. Custom p5 REPRs - MVMP5Interpreter, MVMP5Val, MVMP5Method, MVMP5Package
  4. Custom p5 op - mvmp5::code(string), which takes p5 code and creates an invokable MVMP5Val (wrapping a CV ref)
  5. Appropriate handling (symbol synchronization) of outermost "use" statements to mvmp5::code (not testable until Rakudo (or NQP) is bootstrapped on MoarVM, because the use case of the feature is compile-time from the HLL's parser)
  6. Custom MetaObject implementation (mostly in NQP) for MVMP5Val and MVMP5Package objects (mostly for find_method)
  7. Special handling of p6 type objects as represented in p5, p5 packages in p6
  8. Special handling of HLL-config-ish things of p5 objects in p6, and likewise special handling of override.pm and TIE interfaces of p6 objects in p5
  9. Wiring up the two VMs' GCs by an intermediate references table with refcounts for p6 pointers from p5, and double-increment refcount of p5 pointers from p6, so p6 can hold a reference while also being able to control exactly when the final reference is decremented
  10. calling p5 code (invokable MVMP5Vals that wrap CVs) from p6 (including arg flattening & coercion)
  11. Method calls on p5 objects from p6 (use of MVMP5Method intermediate invokables)
  12. Method calls on p6 objects from p5 (with the optimization of installing stubs into the stashes, a la Moose class finalization)
  13. Integration of CallAsOp module to avoid nested p5 runloops (enabling co-recursive cross VM callbacks in a stackless manner)
  14. p5 wrappers of p6 objects using blessed/tied/autoload SVrefs that all refer to the same dummy SV; indexes to the MVMObjects representing the reference table offsets of each's origin MVMP5Interpreter and referent MVMObject are embedded in another magic on the SV
  15. Marshaling of value types bidirectionally on args to the VMs (P6int/num MVMObjects & IV/NV, e.g.) cross-invocations
  16. Coercion of p5 args to mvm ints/nums/strs registers. These are not [yet] listed in implementation order.

The design of the p5 interop scheme has been thoroughly discussed with Larry Wall, Jonathan Worthington, and Nicholas Clark. A prominent p5p-er offered to help drive any changes needed to p5 to support it, but based on my current design and understanding, no changes will be necessary to support 5.8.3 and later. See http://goo.gl/WLJUQ for one iteration of the working p5interop design doc.

Note: we also considered the JVM while designing this scheme, and nearly all of the design of the MoarVM integration can be directly translated to work on the JVM, even the stackless co-recursive callbacks, since Stefan recently added support for stack saving and resuming for implementation continuations for NQP/rakudo on the JVM. It would probably be another couple of weeks work to port it to the JVM also. Note also that this would provide a more complete bridge from p5 to the JVM via p6 (more complete than Larry's prior work that shipped with p5 for a long time and other Java interop projects).

Project Details

(also see deliverables.) Long term, goals for the p5 interop include well- "productized" (though not commercialized!) tools to package p6 modules as p5 distributions (via MoarVM/Rakudo p6 provided as an extension to p5), the converse (p5 extensions installable to the local p5 module trees via the p6 library management tools), programmatic dual (but integrated) packaging of p6 code for deployment/distribution to both p5 and p6 installations from the CPAN system. The potential exists to use Tobias Leich's p5 grammar & parts of his p5 re-implementation to better facilitate seamless p5 code inline with p6 code as S01 describes, but the modest goals of this project don't include that functionality; this is mainly for loading p5 CPAN modules and scripts into the p6 runtime and accessing their exposed interaction points. The current design also leaves room for p5 ithreads (but not forks.pm) to be supported in the future, but isn't entirely in scope for this proposal.

Project Schedule

I expect the p5 interop to take up to 9 weeks, commencing anytime.

Report Schedule

Progress reports will be mentioned weekly to the #perl6 and #moarvm channels on Freenode IRC, and as such will be published in the logs at irc.perl6.org; certain major milestones of feature availability will get blog posts on the planet Perl 6 syndication feed.

Public Repository

MoarVM's git repository is at github.com/MoarVM, and changelogs for these contributions will mostly exist in the commit logs, as there likely won't be "releases" of MoarVM separately from NQP itself (as MoarVM is fundamentally an NQP backend).

Grant Deliverables ownership/copyright and License Information

MoarVM is published under the Artistic License 2.0, as are these sub-projects. The Perl Foundation will retain copyright of these initial contributions, but not the software as a whole, because patches and fixes to the software won't usually be able to have their copyright assigned to TPF.

Bio

I'm a co-founder of MoarVM, the Perl6-specialized virtual machine. I designed and implemented (or will implement) the strings/IO subsystems, HLL locks and atomic operations, the bytecode validator and disassembler, the QAST->MAST compiler, both VM-internal and REPR-level hashes, the regex compiler, coordination of ending threads and portions of the GC parallelization and work-stealing system, the native and HLL extension system/interface, the threadsafe (and lock-free!) hash and array REPRs and their auto-upgrading from their speedier single-threaded editions, the NFG implementation, the Unicode Character Database character properties and case changes compilation/runtime, and finally the Perl 5 interop scheme.

Amount Requested

USD$10k for 2 months at $5k/month.

21 Comments

+1

I'm sure that diakopter is the right person to do it.

I am a long-time user of Perl 5, and have followed Perl 6 for all these years.

The latest progress on Rakudo Perl 6 is tremendous. This proposal by diakopter is the Grand Unification - if he do it, even if it takes longer than 2 months, it will be money well spent.

+1, because this is IMO The Only Way To Do It(TM), even if that does not sound very perlish. My set goal is it to use Perl 6 at work this year, and with his work, I am sure, this can be done.

+1

This proposal, if well accomplished, is going to be of tremendous help for Perl 6 to become useful more quickly, and for Perl 5 to remain relevant for a longer time.

+1

diakopter has done some seriously good work on moarvm (about 700 commits so far), and has also put serious thought into the matter, so he is about as qualified as one can be for such a task.

I don't think I have to explain to anybody how much both p5 and p6 would profit from such a thing if done right, so I'm all for it.

+1

This is a very exciting proposal.

The Perl6 effort on the JVM has been fantastic to see and now we can have Moar! :-)

Perl5 interoperability will enable companies to build upon their investment in Perl 5 and provide an adoption route for Perl 6.

Go for it diakopter++.

This proposal will see an important and needed task tackled, and I think it comes at a good time. Rakudo's architecture has stabilized a good deal by now, and this comes at the point where MoarVM is in a sweet spot of supporting non-trivial things, but still early on enough to be malleable in the face of this project.

I know that diakopter++ has spent a lot of time considering and researching this topic ahead of making a grant application. And, as the bio states, he has also been involved with MoarVM from the start and responsible for much of what exists today. Overall, there's nobody in a better position to do this work than diakopter.

So, a +1 from me. I'd like to see this work encouraged through funding it, and of course would be delighted to see it succeed. I will be happy to provide any needed input as it proceeds too!

Wooooow that is a lot of money! I have not been looking at the Moar VM for very long, but does it run any Perl-6 code right now?

Does it make more sense to delay this proposal until the Moar VM has proven that it can run Perl-6 code well? That is a lot of money to spend if the Moar VM does not work out very well. Maybe there can be code that someone can salvage in that case, but it still seems like an expensive thing to figure out.

I know my opinion does not matter very much but I am worried that spending all of this money and time right now on something that is unproven will be a big risk. Does it make sense to use the Parrot VM or the Java VM instead, because those can both run Perl-6 code right now?

If there was any project I'd want the Perl community to fund, it's embedding libperl in moarvm, and diakopter looks to me like the right guy to build it out.

But then again, I'd want to see him or other folk incentivized to do any and all work that needs a serious block of focused energy to more quickly get to a solid 6.0.0. I'm concerned that the $100k of Hague money intended to get P6 to 6.0.0 is about spent and we're still, imo, at the current rate of progress, more than a year from 6.0.0. (Fwiw I've thought and said that since 2009, and I'm currently thinking it's more like 2 years away. Imo the only thing that can accelerate it is broad community awareness that several community leaders say it's time to accelerate it.)

I'm thinking there could be a crowdtilt campaign with matching funds from the TPF. Maybe triple matching (one dollar from the Hague dev fund, one from the Hague marketing fund, and another from the general p5 fund). Maybe crowdtilt themselves would contribute another dollar, making it 5x. (Maybe more perl using corporations would chip in to take it even further. Set a cap of $1k max per corporation and I can see 5 stepping in to make this a 10x campaign.)

Then if the campaign can raise $2k from individual contributions (or $1k if there's 10x), we've got the $10k. (Maybe anything in excess of this target flows back to TPF's general fund, so TPF might even come out of this with more money that it starts off with.) This way the TPF and the perl community can send a very strong message of upcoming reunification, and it will have only cost the TPF $4k at most, and the Hague dev fund just $2k, and the perl community can demonstrate support for upcoming P5/P6 reunification.

The JVM efforts and bringing the CPAN content's forward are a breath of fresh air. The kind of work that needs to be done seems hard, and the risk of not supporting efforts like this is bigger than the investment.

Carlos,

Your opinion does matter, and your question/objection is a good one.

MoarVM's future is definitely unclear if you're unfamiliar with the current status of Parrot (especially compared to MoarVM). One thing contributing to this confusion is the fact MoarVM was only recently revealed, when it has in fact been under *intense* development for more than a year prior to that. Here are some facts about MoarVM and Parrot's status w.r.t. Rakudo that should promote some of this tribal knowledge to general awareness.

1. First, MoarVM has many more active developers than Parrot. The last remaining active Parrot developer has publicly said he wants to take Parrot in a new direction (notably, one that doesn't support Rakudo Perl 6). MoarVM has at least 4 extremely active contributors, and another several active contributors.
2. Second, Parrot has technical limitations preventing further expansion of Rakudo Perl 6's capabilities using the Parrot backend, which preclude it from being a good platform on which to prototype Rakudo's libperl interop.
a. Parrot's concurrency features (in their current form) are devastatingly unusable by Rakudo:
1) Its barriers (to intercept accessor and mutator PMC opcalls and redirect those calls to the thread that allocated the PMC) are only on the PMC v-tables, and not on the 6model v-tables, which are what Rakudo uses, so that mechanism would need redesigned in light of 6model, of which the author of Parrot's threads clearly wasn't aware.
2) Currently, only parrot's main (first/original) thread can spawn new threads, and its child threads cannot interact with each other via shared objects, so objects returned to the main thread from child threads should not be accessible from another child thread [... but there is no built-in protection against this]. Claims this is easy to fix are less than veracious.
3) Since only the allocating thread can access or mutate its shared objects, no object can be accessed by more than one thread at once, so there can be no such thing as lock-free shared objects, since every object is thread-safe, due to requiring a lock for all operations, including reading! However, it is even worse than this: all objects created by the main thread *share the same lock*, since only one of them can be accessed at once. This manifests practically as a limitation that if you want to have multiple child threads utilize shared objects, the only work that can be done concurrently is work that doesn't access any shared objects. This relegates Parrot threads to the same effectiveness domain as JavaScript's web-workers.
4) Parrot's garbage collector has no way of knowing when a given object can be freed, because it does not know whether or not it has passed that pointer to another thread (deeply accessible/nested), if there are no more references to that pointer in its own thread. Parrot's thread design attempts to mitigate this by keeping track of which objects it *explicitly* sends to other threads... but it fails to keep track of any objects reachable from *that* object (at the time of passing it), if that pointer is a collection/container object with links of its own, but then the link to the other object was broken, but is still held as a reference by another object, in that other thread. This effectively means that no memory can ever be safely freed by Parrot, and thus it has no garbage collector. How someone thought one could invent a safe concurrent garbage collector with only a few hours work is pretty much unfathomable, especially when one is discovering an entirely unheard-of method of doing something that normally takes thousands of hours of work by professionals with decades of related experience, instead of being designed for a junior undergrad semester project.
b. Parrot has no asynchronous I/O, and no one has any plans to incorporate it, contrary to the Parrot Design Documents, which are written as if it already exists. MoarVM has resources committed to furnishing this by this year.
c. Parrot has no knowledge of anything having to do with signals (and again, no one has claimed they are available, willing, or able to provide them); MoarVM will have them soon.
d. Parrot has no interaction with (or control of streams of) child processes, except that it can read stdout from a child process (and again, no one has claimed they are available, willing, or able to provide anything beyond this). MoarVM has somewhat more control (via APR currently), but when APR is replaced (probably by some variant of libuv), we'll enable capabilities equivalent to Perl 5's child process extensions.
e. Parrot has no atomic operations; MoarVM does.
f. Parrot's potential for optimizing Perl 6 regexes is very limited due to its reliance on libicu and its limited API. MoarVM already has its own implementation of the Unicode Character Database, and so doesn't have this limitation.
g. Parrot has no JIT (and again, no one has claimed they are available, willing, or able to work on this). MoarVM plans one within a year (and again, more than several qualified resources are lined up to do it).
h. Parrot takes ages to build compared to MoarVM (and not because it provides more features), which constrains its developer productivity dramatically.
3. Even parrot devs (3 recently core/chief ones, afaik) have said MoarVM can be viewed as a continuation of Parrot's ideals and goals... implying they view Parrot's ability to further those goals has reached its end [recently, if not a long time ago].
4. At this moment, MoarVM passes over 90% of NQP's test suite (NQP defines the opcode set for Rakudo backends and a Perl-6 like language with many of its features). The JVM backend accomplished this only a few short weeks ago, and now passes over 99% of the Perl 6 test suite that Rakudo on Parrot passes, which bodes well for MoarVM's ability to catch up quickly.

As stated in the grant application, much of the work is already applicable to the same features on the JVM backend; in fact I considered proposing funding that instead of using the MoarVM backend, but I chose MoarVM when it became clear Rakudo would likely be caught up in time.

Regarding the amount of money, it is about 3/4 of the remaining uncommitted funds and 1/2 of the remaining total funds. Much (or all?) of the $100k portion of the Hague Grant was for fund-raising activities, and that still remains. Do you have any ideas for how to spend that, other than hiring someone full-time for a year to fund-raise (which was previously proposed and rejected)? [I can't think of anything right now, but I can think of plenty to do once Perl 6 has a marketable product.]

I appreciate your question; I hope this answer has helped clear up some of the cognitive dissonance.

LOL you didn't answer my question! You gave a laundry list of things that Parrot doesn't do and that you hope the Moar VM will do in the future.

I have also embedded libperl, and I didn't need asynchronous I/O, JIT, or threads to make the design work. Maybe I will offer to do this work for $8000 USD and save the TPF some money.

Carlos,
$10000 isn't that much. If what you really care about is saving TPF some money, you may could sponsor TPF $2000, or you can do this interop work for free. Note that the work will be not that easy(mostly will be harder), because that need many changes to MoarVM itself in a spiral development mode, that means diakopter++ may spend more time than two months. Actually, if things go well, the value of the grant work will be much more than $10000.

"you didn't answer my question" - I answered each of your three questions:

You asked whether it runs Perl-6 code; I said: 90% of NQP's test suite, and NQP is a large subset of Perl 6.

You suggested waiting until MoarVM runs Perl-6 code well; I answered with an estimate of when it would catch up to Rakudo-JVM. But even if it didn't, did you forget "run Perl 6 code well" is included in (a prerequisite of, in fact) the grant being completed, and that TPF doesn't pay the grant until it's completed? There have been plenty of proposals that were accepted, but not completed, and therefore not paid.

You suggested it would be better to use the Parrot VM or the Java VM instead, so I explained why Parrot wasn't a good choice (since Rakudo is blocked from progressing on it), and also why I didn't choose the JVM.

"I have also embedded libperl" - If you embedded libperl in a Rakudo implementation, with the degree of interoperability proposed, and you're willing to contribute it, feel free to submit a pull request; we'll gladly review it.

"and I didn't need asynchronous I/O, JIT, or threads to make the design work." - I didn't say I needed those things to make the design work. I was saying Parrot has no plan to implement those things, so its long-term viability as a Rakudo backend is in question, so it's not a good choice for this project.

+1

I have followed MoarVM since before it was revealed, and also been aware of diakopter's plans to get good, reliable interop on top of the VM. I know he has spent a lot of time thinking things through, discussing with various other people, and planning his approach.

I believe diakopter is the person to do this work, and I am excited to see this grant proposal. It has my full support.

10.000K for blitzkost on MoarVM? Why not. If we have the money, use it.
But beware that it will not lead to a faster perl5, just better p5 integration into p6. As blitzkost did for parrot/rakudo.

There are several false points in the discussion/limitations on parrot threads, but I'm happy that the current opinion on parrot threads are finally public somewhere. And this has nothing to do with this proposal.

+1

+1

Making this work would remove a big barrier to greater Perl6 adoption, and I heavily support the concept.

John Napiorkowski

+1

I think the proposed schedule is optimistic, but that's certainly no reason to preclude funding this proposal.

And if anyone can do the work, it's diakopter. He has the knowledge and experience necessary to accomplish the goal of attaining perl 5 interoperability on moarvm. So, again, +1 for funding.

Whatever you say dude! I think you told me why you want to work on your vanity VM instead of working on something that already works.

I think I liked this proposal better when it was called Blizkost, but I can see now that Perl-6 is one long farce of throwing away projects before they are done because it is always more fun to start something new. I am glad it is not my money TPF is going to throw away here.

I think that this work is worth funding.

I am somewhat troubled that as-is the grant proposal will mean that diakopter might not get paid despite delivering code that meets all the goals. That would be silly, and should be avoided.


Firstly, is the intent to pay all $10,000 when the project completes? Or is the plan to have a halfway payment of $5000 after (projected) one month's work? Because I'm doubtful that the 16 items listed on "feature details/schedule" are of equal size, and in a linear order. Hence I don't think that items 8 and 9 mark the halfway point. What does?


Secondly, is it specifically required to support Perl 5 all the way back to 5.8.3 to complete the grant, or is that just a "nice to have"? There have been a lot of bugs fixed in the 9½ years since 5.8.3 was released, and it may turn out that some of them are essential to making interworking work. I remain of the opinion that it's more important to get completeness, correctness and breadth of support on current Perl 5 (be it 5.18.x or even 5.20.x) than divert resources to working around bugs and problems in older versions. Support for older versions of Perl 5 is nice to have, but shouldn't come at the cost of reducing functionality on current versions.

I don't think that restricting support to more recent versions will hurt uptake. Until distributions pick up MoarVM to package, people wanting to use it will have to build it from source, which means that they are also capable of building a current Perl 5 from source. By the time MoarVM is packaged by distributions, those distributions will also be packing a current Perl 5.


Thirdly, are all the 16 detailed features required to be delivered and working in order for the grant to be paid? If it turns out to be too hard to get some working correctly in the timeframe, but the rest of the features are working and the project delivers production grade interoperability suitable for a wide range of uses, has the grant failed? I think not, but it's not clear from the proposal how to assess this.


Is there an acceptance test for the project? Something like having a web framework running in Perl 6 accessing databases via DBIx::Class and DBI running on Perl 5? Calls from Perl 6 into methods written in Perl 5, passing callbacks to Perl 6 routines (and vice versa). Demonstrating Perl 6 classes derived from Perl 5 classes in turn derived from Perl 6 classes?


Finally I think that MoarVM is the right VM to target first. As both MoarVM and the Perl 5 interpreter are written in C, it will avoid additional problems in having to bridge from Java to C, which would complicate targeting the JVM. I don't think that targeting Parrot first is a good idea, as it's not clear whether Parrot will even be supporting Perl 6 in the future. See the thread starting at http://lists.parrot.org/pipermail/parrot-dev/2013-June/007489.html (Although it must be noted that to date there is no consensus on the direction Parrot will take.)

+1

I feel strongly this is an important direction for Perl 6, and have a great deal of confidence that diakopter is the right one to pull it off.

Leave a comment

About TPF

The Perl Foundation - supporting the Perl community since 2000. Find out more at www.perlfoundation.org.

Recent Comments

  • Solomon Foster: +1 I feel strongly this is an important direction for read more
  • Nicholas Clark: I think that this work is worth funding. I am read more
  • Carlos del Rey: Whatever you say dude! I think you told me why read more
  • perlpilot: +1 I think the proposed schedule is optimistic, but that's read more
  • John Napiorkowski: +1 Making this work would remove a big barrier to read more
  • Reini Urban: 10.000K for blitzkost on MoarVM? Why not. If we have read more
  • Carl Mäsak: +1 I have followed MoarVM since before it was revealed, read more
  • diakopter: "you didn't answer my question" - I answered each of read more
  • JimmyZ: Carlos, $10000 isn't that much. If what you really care read more
  • Carlos del Rey: LOL you didn't answer my question! You gave a laundry read more

About this Entry

This page contains a single entry by Karen published on July 24, 2013 7:35 AM.

Next Release of Pinto With Key Features - Grant Report #1 was the previous entry in this blog.

Improving Perl 5: Grant Report for Month 19 is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

OpenID accepted here Learn more about OpenID
Powered by Movable Type 4.38