Grant Application: Improving Cross compilation of Perl 5
Tue, 01-May-2012 by
Karen Pauley
edit post
<![CDATA[
We have received the following grant application under the "Perl 5 Core Maintenance Fund":http://www.perlfoundation.org/perl_5_core_maintenance_fund from Jess Robinson.
Before we vote on this proposal we would like to get feedback and endorsements from the Perl community. Please leave feedback in the comments or send email with your comments to karen at perlfoundation.org.
**Name:** Jess Robinson
**Project Title**: Improving Cross compilation of Perl 5
**Synopsis**:
Perl 5 has several scripts and tools for cross-compiling it for other platforms. These are somewhat inconsistent and scattered across the codebase. My aim is to complete a cross-compilation target for Android, with smoker, and to prepare and document ways to add more targets without specialising for each of them.
The implementation goal is to allow direct cross compilation (no changes to actual Perl C-code, just using the hints system and configuration of cross-compiler paths and flags) to as many current and future platforms as possible, without needing knowledge of the build system internals. Ideally this will automate as much as possible the collection of data for the config files. Only platforms on which we can't run arbitrary compiled C code (eg Android) should get pre-defined config files in core, and if these become unmaintained (not updated when new features are added to Configure) they should be flagged as stale.
**Benefits to Perl 5 Core Maintenance:**
This work will remove duplicate build scripts and "specialised" code created for various cross compilation environments and either them replace with a generic version, or leave them out completely. Either way this will allow Core contributors to have a better overview of which specialised code needs to be maintained.
**Deliverable Elements:**
I propose to work this as a pay-as-you-go grant, similar to Nicholas and Dave. The initial request is for 200 hours at $50/hour, over 2 months. If the project proves to need less time than this, we may consider it done and end early, with the payment covering the actual number of hours worked. Alternately, if it requires more time I would be willing to work longer at the same rate to complete unfinished tasks.
The actual deliverables will be determined by the project, but will cover at least the delivery of a working Android cross-compilation build environment, together with a smoke environment to ensure that it continues to compile.
I will deliver weekly summary reports on progress via the p5p mailing list, and an end of month report summarising the whole month. I would then claim for the hours worked in that month.
The grant managers would be allowed at any time to inform the board that the project is failing, and the TPF board, after allowing me to present my side, decide to terminate the project.
**Project Details:**
I intend to spend time on the following items:
1. Bring my existing Android cross compiling environment and copy of Perl up to date and working. The current branch is based on perl 5.15.1, records work in progress, and uses an NDK two versions out of date. The branch needs rebasing and some restructuring to eliminate extraneous intermediate stages. The version of NDK used doesn't upgrade cleanly to newer ones. Fixing this will require trying to make the use of the NDK more generic and easier to configure in case of more path/binary renaming in future versions. approx 50 hours.
2. Set up a smoker for Android cross-compilation to ensure that the build continues to work. - 10 hours.
3. Refactor the working Android cross compiling approach to minimise divergence from the regular hosted build process, simplifying it as much as possible.
In particular:
* fully exploit the fact that it now uses ./Configure -Dmksymlinks to separate the host and target build trees, which wasn't done previously.
* use lib/buildcustomize.pl to simplify the cases where the host miniperl driving the build needs to report configuration values for the target system.
* assume that a functional /usr/bin/perl is present when cross-compiling, and convert top level driver scripts from /bin/bash to Perl for easier maintenance and improved portability.
[Flexible guess. - 30 hours]
4. Identify a second target platform to cross compile to, representative of a small consumer device running Linux or *BSD on a non-x86 chip, and which can be run under emulation for testing. One plausible candidate is the Raspberry Pi, an ARM Linux device, because developer virtual machines exist, are documented and actively maintained. Get cross compilation to this device working, using the toolchain documented in Cross/README.new, and also using the approach documented in INSTALL. - Unknown.
Refactor the cross compilation code described in the previous steps to
* take advantage of the simplifications possible with -Dmksymlinks and buildcustomize.pl approaches used by the Android port
* minimise the amount of duplication and overlap with the regular hosted build process
* share with the Android port as much code as is practical
* maximise maintainability
* aim toward an approach that where possible is generic, minimising the use of pre-requisite "canned" config files.
5. Document this with a "Howto cross-compile Perl" guide.
6. Set up a smoker for cross-compilation to the small-unix target to ensure that the build continues to work. - 10 hours.
7. Review the various Cross compilation scripts and tools existing today in the core code, to determine which are still required using the revised approaches above. Tidy up by removing / deprecating un-used or obsolete parts.
**Stretch goals:**
1. Devise and implement a way to perform an installation of the resulting build onto the target smoke platforms (using an emulator), run the perl 5 core test suite, and gather the results to report back to the current smoke system. - Unknown
2. Investigate ways to cross-compile CPAN modules, especially XS modules, and install onto the target device. Likely this would be several steps, starting with modules that don't probe the local system within their Makefile.PL, and build them as part of the main cross build. Then work to decouple building modules from building the perl core, possibly including making the cross-building setup installable on the host machine. - Unknown, estimate 25 hours.
I estimate that the main items 1 to 7 will take around 200 hours to complete. I will work on each item in turn (though not necessarily in the above order) and in stages to provide usable patches during the project.
These items were prepared with much help from Nicholas Clark, many thanks to him.
**Project Schedule:**
I expect to be working on this alongside other work, and realistically to be spending about half my time on it. Thus I am asking for 200 hours over 2 months.
I am available to start work on this immediately.
**Bio.:**
I have worked with Perl professionally for the last 7 years, and I spend much of my spare time supporting and encouraging the use of Perl.
I took the Perl 5.10 changes and additions from the android-scripting repo and ported them to enable cross-compiling of Perl 5.15 for Android devices last year, this work can be seen on github: "https://github.com/castaway/perl/tree/x-compile-android":https://github.com/castaway/perl/tree/x-compile-android.
**Endorsed by:** Nicholas Clark, Ricardo Signes
**Amount Requested:** $10,000
]]>
Comments (9)
Thank you, Jess, for this grant application. For what it's worth, I'd fully support it.
+1
Sounds like a great project to increase perl support for awkward embedded environments, of which there are many. Even if you have a full Linux install, you can be way short of ram to compile with optimization.
This sounds great -- but I think this task should rather be performed by Dave Cross!
At this time cross-compiling perl is painful, so much that we prefer to compile it directly on a veeerrrryyyyy slooooooow virtual host...
So YES, perl need to be cross-compiled, because of course perl can run in embedded systems, it is very stable and its memory consumption is quite reasonable, even if it could be better... Another specific grant perhaps? :)
Perl has languished in this area for many years and it would be nice to finally get proper cross compilation support. Jess has already worked in this area and I would be pleased to see her given the opportunity to put some serious effort into the problem.
Given that Perl 5 has had broken cross compilation support for years, this looks like one of those thankless, awkward tasks that enthusiastic volunteers shy away from fixing. I welcome TPF funding such work.
Many operating systems use Perl as a core component on various hardware platforms. Supporting the cross compilation of Perl has clear benefits to these users.
Finally, although I don't know Jess very well, she strikes me as someone who will do a good job of this task.
I fully endorse this grant application.
I'm very keen to see this work happen. There's the obvious direct benefit of providing Perl on Android, which is turning into a very popular platform, hence somewhere that it would be good for Perl to be represented on. The proposal also addresses a longer term problem - cross compiling is a fairly frequently requested feature, but each time the desire is to build for a different embedded target platform. Automating much of the process of cross compiling to small embedded systems would give us a more generic system which is future proof (as far as that is possible) for whichever of the many possible OS & CPU combinations we're yet to be asked about. Finally, having a known working system, and confidence about which platforms and classes of platforms it will support will allow us to clean up code related to previous attempts at implementing cross compiling which targets those same platforms, without fear of accidentally eliminating the functionality.
I think that Jess is an excellent candidate. She has already demonstrated both the ability and the desire to get Perl running on Android by getting perl 5.15.1 cross compiling to it on an older toolchain. I think she's the right person to fund to deliver the goals of the grant, and thereby get us to working and maintainable *nix cross compiling.
Perl for Android is what I'm waiting for. Good luck to you Jess!
