October 2013 Archives

Tony Cook has requested an extension of $13,000 for his Maintaining Perl 5 grant.

This grant was started in July 2013 and was successfully completed. The requested extension would allow Tony to devote another 260 hours to the project. The funds for this extension would come from the Perl 5 Core Maintenance Fund.

Tony provided detailed monthly reports of the work he completed and these can be found in the following blog posts:

July 2013 Report
August 2013 Report
September 2013 Report

Before we make a decision on this extension we would like to have a period of community consultation that will last for seven days. Please leave feedback in the comments or, if you prefer, email your comments to karen at perlfoundation.org.

Nicholas Clark writes:

As per my grant conditions, here is a report for the August period.

Multiple cores are the future! And we need to enjoy this, because we aren't going to get any choice about it. The current hardware for the hot backup for perl5.git.perl.org has 24 cores, and even mobile phones are thinking about going quad-core.

The upshot of this is that the more that you can get the computer to do parallel, the faster the entire task will complete. On *nix the make has been able to run in parallel for quite a while, and back in 2008 I hacked away on TAP::Harness until we got the tests running in parallel too. (Sadly only on *nix - Microsoft's divergent implementation of socket() frustrates the same approach being used on Win32.)

Well, to be strictly accurate, the make has been able to run in parallel except when it can't, due to race conditions we were previously unaware of. (Just like the tests, actually.) We chip away at them as we find them, and I guess that the best we can hope for is "no known problems", and an increasingly long time since the last "unknown unknown" turned up.

Frustratingly there had been one "known problem" that we hadn't figured out a way to solve. SDBM_File contains a subdirectory in which is the source code to libsdbm.a, and the Makefile.PL for SDBM_File ends up generating a Makefile which has two rules, both of which recurse into ext/SDBM_File/sdbm/ and try to build the same target. Run in parallel, and you can end up with one process deleting libsdbm.a at just the same time that a second process is trying to link it, resulting in that second process bailing out with an error and the build failing. One rule was the default "build my subdirectory" rule generated by ExtUtils::MakeMaker. The second was a custom rule generated by the Makefile.PL to say "to build the dependency libsdbm.a, you need to do this..." and the difficulty was that the rule that we needed to keep was the second one.

What has eluded all of us for a very long time is that actually it's pretty easy to stop ExtUtils::MakeMaker generating that default rule - if you know how. So the fix turns out to be quite simple, once you know the secret, and now we think that this bug is finally fixed, and as no others have cropped up in the Makefile for a while, it's starting to look like the Makefile might be bug free and reliable in parallel.

Jeffrey Ryan Thalhammer reported:

This past May, The Perl Foundation awarded a grant to fund development of a couple features in Pinto. Pinto is a robust tool for curating a private repository of CPAN modules, so you can build your application with the right modules every time. This is my second progress report on that work.

The next deliverable on the grant proposal is a merge command. The idea here is to merge the indexes of two stacks, much the same way that you merge two branches of source code. This would allow you to create a separate stack for experimenting with additional or upgraded modules, and then incorporate those modules into the master stack once they have been tested and blessed.

I have completed a prototype that handles simple cases when the head revision of stack is a direct descendant of the head revision of the target stack (this is what Git calls a "fast-forward" merge). However the more complex case is when the two stacks have both diverged from the initial branch point. The stack is basically an unordered list, so merging them is not quite the same as merging lines of source code.

There has been a lot of discussion about how this should work on Github. You're cordially invited to give your opinion over there. I'm hopeful that we can come up with a "smart merge" that is specific to the task of managing Perl modules.

Nicholas Clark writes:

As per my grant conditions, here is a report for the July period.

The main theme for this month seems to be "clean up parts of the build".

perl was first developed on a Unix system, back in the times when there dozens of different Unix variants. Hence portability was initially across the different C libraries (and even C compilers - remember, Perl predates the C89 standard, let alone its implementation). Figuring out precisely what the system could(n't) do (correctly) was performed by a shell script, Configure, and it expanded its variables into files needed for the build by using simple shell scripts as templates.

It's still how configuration is done by perl 5, and "figure things out with a shell script" is the standard way that pretty much all open source software builds on *nix systems. There's an awful lot of useful hard-won knowledge distilled into these shell-based configuration systems, and whilst they aren't that pretty, they work, which to the end user is what matters.

The template expansion is necessary because to maintain sanity, it's necessary to maintain a clean distinction between files that are generated as part of the configuration and build, and files that shipped with the package (from a tarball, or from a version control system). There's implicitly a concept of who "owns" the file - the human using a text editor, or the build system.

Makefiles are one of the types of files whose contents need to vary based on things determined by Configure. Hence, it naturally falls out from the above description that the various subdirectories have Makefiles which are generated by templates implemented as shell scripts. This plan doesn't fare so well now that "portability" no longer just means to different *nix systems, but also to systems that don't even have shells.

A particular mess was the Makefile for the utils/ directory. Like the other Makefiles it had been being generated by running a Makefile.SH. Unsurprisingly this doesn't work on Win32, so the solution was to also check the generated Makefile into the repository. Just to add to the fun, here VMS is special, and has rules in its top-level Makefile (DESCRIP.MMS) to generate files in utils/, and completely ignores utils/Makefile.

This particular Makefile has very few configuration controlled changes, so typically the regenerated version was identical to the checked in version. Unfortunately for some configurations it would differ, with the result that configure && make of a clean tree would result in a dirty tree, with (seeming) unchecked in changes. This is wrong - building should never make a clean tree dirty.

The irony is that the Makefile generation doesn't need to be a shell script. By the time that utils/Makefile is needed, there is already a fully serviceable miniperl ready and waiting.

So I bit the bullet and replaced utils/Makefile.SH with utils/Makefile.PL Carefully. Firstly by replacing the shell script with a perl script that generates a byte-for-byte identical utils/Makefile (including boilerplate that claims it was generated by utils/Makefile.SH), which was called from the same place that the *nix Makefile had been calling Makefile.SH

About TPF

The Perl Foundation - supporting the Perl community since 2000. Find out more at www.perlfoundation.org.

About this Archive

This page is an archive of entries from October 2013 listed from newest to oldest.

September 2013 is the previous archive.

November 2013 is the next archive.

Find recent content on the main index or look in the archives to find all content.


OpenID accepted here Learn more about OpenID
Powered by Movable Type 6.2.2