Grant Proposal: Automated generation of DWIM Perl

6 Comments

We have received the following grant application. Please leave feedback in the comments field by March 22nd, 2014.

Automated generation of DWIM Perl

  • Name:

    Gabor Szabo

  • Email:

    undisclosed

  • Amount Requested:

    $6,000

Synopsis

DWIM Perl is a "batteries included" Perl distribution for Microsoft Windows and for Linux. On Windows it is based on Strawberry Perl, on Linux it is compiled from the source released by the Perl 5 Porters. Both versions include a few hundred additional modules from CPAN.

The grant will enable me to further work on it, and specifically to finish automating the release of new versions.

Benefits to the Perl Community

DWIM Perl can make it much easier and faster for beginners to get started using Perl. It is useful both for people who would like to program in Perl, and for people who "just" want to run existing scripts or applications.

DWIM Perl makes it much easier and faster to set up a server running a Perl-based application. For example on a $5/month VPS. This reduces the advantage other languages might have in the area of deployment.

DWIM Perl can be the foundation of a "Perl Cloud", where people can develop on either a Windows or Linux based machine, and then easily deploy to a cheap VPS that has the same version of Perl with the same set of CPAN modules.

The grant will allow me to cerate a tool for automated releases and semi-automated upgrade of the parts. This will allow me (or anyone else) to easily create a new, up-to-date release of DWIM Perl.

It will also make it easy to create similar "batteries included" Perl distributions for the needs of companies, or open source projects using Perl.

Deliverables

  • A script that given a configuration file can build a DWIM Perl distribution for Linux and for Windows.
  • Configuration files ready to build DWIM Perl for Linux using 5.18.2, and for Windows using Strawberry 5.18.2.1.
  • A new release of DWIM Perl for Windows based on Strawberry Perl 5.18.2.1 using a scripted generation.
  • A new version of DWIM Perl for Linux based on 5.18.2
  • DWIM Perl for Linux based on 5.19.X.
  • Bug reports or even patches to modules that don't install cleanly.
  • All the source will be placed on GitHub.

The Windows version will be an installation process. The Linux version will be a tar.gz file to be unzipped.

Project Details

In order to get started using Perl with any reasonable modernity, people need to find and install hundreds of CPAN modules. This is an additional obstacle to getting started with Perl. Not only can it take hours to install all the required modules, there is always a chance that the current version of some of the modules on CPAN do not install cleanly. Several important CPAN modules require additional, non-perl packages. Locating and installing them can be additional issue.

Having a Perl distribution with "batteries included" means that newbies can download, install, and run. A lot less friction in getting started with Perl.

Each DWIM Perl distribution will include

The Windows version will also include wxPerl and Padre.

The Linux version will include Starman.

Inch-stones

On Linux, there is already a relatively good script, located at https://github.com/dwimperl/dwimperl-5-18-linux, but there are several modules that require external dependencies, that are not included.

In the Windows version, Strawberry already includes these "problematic" modules so the main thing that is missing is the automation.

For both versions the handling of the stable CPAN snapshot is missing.

  • Research how the CPAN snapshot should be handled: Options that should be checked are: Carton, Pinto
  • Do a "market research" asking people for lists of prerequisites they have in their Perl-based applications. Both open source and closed-source applications. This will provide valuable input as to what needs to be included in DWIM Perl.
  • Create a new release of both versions using a simple script, and with one module in the CPAN snapshot.

    The Linux script will be based on https://github.com/dwimperl/dwimperl-5-18-linux

  • Add tests that can be run post installation to verify the problematic modules work. (Mostly the ones that require external libraries.)

  • Write script to update the sh-bang lines of the perl scripts after relocation. (This might be submitted to p5p as well.)
  • Include DBD::mysql in Linux.
  • Include DBD::Pg in Linux.
  • Include POE in Linux.
  • Include PAR::Packer in Linux.
  • Include XML::Parser in Linux.

There will be several intermediate DWIM Perl releases, marking the finish of each inch-stone.

Project Schedule

I can start immediately after the grant is accepted on March 31, 2014. I expect the project to be done within 2-3 months.

Completeness Criteria

Assuming the evaluator has access to the right operating system (MS Windows or Linux) she will be able to run the build script and create a new version of DWIM Perl with minimal manual intervention.

Bio

I have created the DWIM Perl project in 2011, originally as a vehicle to distribute Padre, the Perl IDE on Windows. Later the Linux version was added in order make it easy to deploy Perl-based applications on Linux-based servers.

I have been using Perl since 1995 and teaching it since 2000. I am the original author of Padre, the Perl IDE. I am also the maintainer of a number of modules on CPAN https://metacpan.org/author/SZABGAB.

In addition, I run the Perl Weekly newsletter and the Perl Maven site.

6 Comments

Please support this. Dwim Perl is a very handy ‘everything included’ distribution of Perl it's useful to be able to point beginners at, or ask Windows sys-admins to install for supporting Perl applications.

It would be great to have this updated from v5.14.

And Gabor has a proven record in the Perl community of getting things done. The proposal seems well-thought-through, so it seems likely to succeed as planned.

This would be a boon to the Perl community; please support it.

I think DWIM perl is a great idea and I have great respect for Gabor's work to evangelize Perl in many forms.

However, the overall size of the grant request seems excessively large for the effort involved.

I was the original creator of Strawberry Perl, including its automated build tools. It was implemented over the course of a YAPC (based, admittedly, on the Vanilla Perl contest run by Alias previously).

As I expect satisfying Linux dependencies will be easier than satisfying Windows ones, I don't see why that part of the effort would be much larger. The other inchstones seem relatively straightforward.

My general impression is the same as David's. This is not a bad idea, but I don't think the money matches the necessary work, nor do I think it's the best spending of the limited community funds we have available.

I look at this grant and my first thought is that $6000 is equivalent to 120 hours of Tony Cook or Dave Mitchell. Much that I'd like to see this grant delivered, if funding this grant prevents Tony's open application from being renewed, then I think that TPF's prioritisation is wrong.

I have more specific concerns about the proposal.

$6000 is a lot of money. At the below-commercial rate that P5CMF has been paying grants, that equates to 120 hours. Is there really 3 weeks' full-time work needed for the deliverables here?

The work described is just to set things up for the first release. How is ongoing maintenance going to be handled? Experience both from core development and installing dependency trees from CPAN is that you can't assume that integrating a bunch of modules will work perfectly every time. The proposal acknowledges this too - there is always a chance that the current version of some of the modules on CPAN do not install cleanly. Upgrading one module can cause another's test suite to fail. Module upgrades may be intentionally incompatible - documented changes to an API. An end-user has the choice of whether to make such an upgrade, based on their individual use case. A distribution has to be one size fits all. The plan lists a lot of modules - the problems are likely to scale as O(n²) with the number of modules, not O(n). So this could become a big pain. If DWIM Perl can't kickstart without a grant, how will it avoid becoming "special biologist word" without further grants?

I see that Task::Kensho is listed. My understanding is that Task::Kensho lists "current best practice", and will drop modules from its list of recommendations if a better module emerges. This makes sense for it. But for DWIM Perl, what's the deprecation policy for removing modules? If someone builds their business codebase on DWIM Perl, can they upgrade safely to get bug fixes and other improvements, or will they run the risk of modules they were using being pulled from under them because Task::Kensho dropped them? Or is the plan for DWIM Perl to be a one-off? You say where people can develop on either a Windows or Linux based machine, and then easily deploy to a cheap VPS that has the same version of Perl with the same set of CPAN modules which would imply that there will only be one release of DWIM Perl ever. Otherwise, how would such a deployment work, as differing DWIM Perls would have different module versions.

You state A large part of Task::Kensho. Which modules do you think you will need to omit, and for what reasons? It's really not clear from the proposal, and it's hard to evaluate without more clarity on this.

Why all of Catalyst, Dancer and Mojolicious? (and hence all their dependency chains). This strikes me as a lot of work for little gain. You can't please everyone, so don't kill yourself trying. Surely the aim of this distribution is to be a reasonable out-of-the-box set of defaults for people who aren't aware of the choice on CPAN, so why the need to offer (and confuse) with choice here?

While Do a "market research" asking people for lists of prerequisites they have in their Perl-based applications. Both open source and closed-source applications. This will provide valuable input as to what needs to be included in DWIM Perl sounds laudable, I fear that it won't generate an actionable list. A lot of people will have a lot of different ideas, likely contradictory, and likely suggesting multiple modules that perform the same task. Also, is this item that costly to set up? It would have made a much better grant proposal if this had been run before applying, so that the project scope would be clearer.

You mention that several important CPAN modules require additional, non-perl packages. Locating and installing them can be additional issue and you specifically list DBD::mysql, DBD::Pg and XML::Parser. Are you proposing to bundle source for expat and for the MySQL and PostreqSQL client libraries? Or are you going to assume the same behaviour as core perl (and Python and I believe PHP) - if the C libraries are not present, the relevant extension is not built? Each of yes and no will lead to pain, but in different places.

By Linux do you mean any architecture that Linux builds on where Perl, expat and the databases run? Debian runs on more than 10 architectures, Fedora on at least 6, and perl will build on all of them and more. Or did you just mean x86 and x86_64?

The whole proposal strikes me as a highly ambitious "big bang" approach to trying to solve what is a legitimate problem. It would feel a lot less risky, and I'd be a lot more comfortable, with a proposal that started smaller and iterated based on feedback and experience. For example, start by taking just Task::Kenso and producing a distribution that builds on Win32, current CentOS and current Ubuntu, and see how that goes.

DWIM Perl looks like an interesting idea.

Whereas most grant proposals request funds for a project of fixed scope, keeping DWIM Perl up to date seems more like an ongoing operational matter.

DWIM Perl's existing Github repositories show that so far only Gabor has contributed to this interesting idea.

As time passes, DWIM Perl will need to keep up to date with new releases of Perl, CPAN modules and the supported operating systems.

For an ongoing operational task like this, I would expect to see more people involved so that work can continue if the maintainer becomes busy. Perl 5 itself and large CPAN modules all take such an approach.

I support the ideas behind DWIM Perl, but I consider funding its one developer for such a large amount at this stage a risky proposition. Having more contributors developing and releasing this project will significantly reduce that risk.

I understand people are worried about the size of the grant. After all previously these grants
were limited to 3,000 and only recently was the limit raised to $10,000. As I undersand
this was done in order to allow bigger projects without cutting them up in small pieces.
The only bigger grants were the ones given to Nicholas Clark, Tony Cook, and Dave Mitchell,
but as far as I know those come from a different budget.

Anyway, let me try to respond to some of the concerns raised here.

@David Golden, if I understand you correctly you imply that this whole things should be a few hours work or a few days at most. I understand that you could do this during a YAPC but I doubt I can do what I planned for DWIM Perl in such a short time, in a way that will be maintainable long-term.

@Nicholas Clark, if you have checked out the already existing releases of DWIM Perl, you probably saw that there were releases of both the Linux and Windows versions. They were partially automated, but still a large part was maual work.

Module upgrade is indeed not straight forward. That's why I need to make sure that the back-end I select (Pinto, Carton, etc.) can handle fixing module versions and easy upgrade and downgrade. This will still require manual work from me, but far less than it is today, that will allow me to handle the ongoing maintenance on a volunteer base in my free time.
My experience with the previous releases is that there are not too many cases when I can't just upgrade to the latest in most distributions.

I am not sure what do you mean by "If DWIM Perl can't kickstart without a grant, ...". First of all DWIM Perl exists, so it has kickstarted. Second, do you actually say that in order to get a grant I need to prove that I don't need a grant?

Task::Kensho the general guideline is to include everything that can be installed on the platform.

Backward compatibility and deprecation policy: There is no commitment to backward compatibility. If a module stops working it will be removed from DWIM Perl. Otherwise they can be kept in the distribution as including them does not incur a big cost. Modules might change in a backward incompatible way, but DWIM Perl is not committed to maintain backward compatibility.

OTOH the whole point of the grant is to build to a tool that companies can use to easily build their Perl distribution consisting the standard perl + a bunch of CPAN modules at specific versions including external dependencies. They can then decide to either go with DWIM Perl or build their own distribution.

Why all of Catalyst, Dancer and Mojolicious? - while reducing the need to choose can certainly help beginner, the primary objective of DWIM Perl is to eliminate the need to install. OTOH having only those 3 is already a big reduction in choices.

Market research: I am not sure how can the idea of what to include be contradictory? There are no mutually exclusive modules on CPAN, are there? Also, market research is not the same as asking for an order. In the end I'll have to make a decision what to include in each release. Oh and before someone misunderstands, DWIM Perl already hase a nice user-base. So there is no doubt that it already serves a market.


External dependencies: I am planning to include the already compiled part of the MySQL and PostreqSQL clients along with expat and other dependencies. Please check bootstrap.sh on how it is being done for some other libraries.

By Linux I just mean x86 and x86_64. At least for the grant.

@Tom Hukins, besides p5p, which other perl project has multiple developers?

As I understand Strawberry Perl always had one maintainer. DBIx::Class, Catalyst have minimal contributions outside the main developer. This seems to be how most of the Perl-related projects operate.
BTW The grant is requested to make it easy for anyone to build DWIM Perl and thus either to contribute or to take over.

TPF is going to pay any money only after the project has finished successfully. So the only risk is that TPF allocates the money for a few month. If after a few months the project has not succeeded, the money is freed again for other grants.

As far as I know TPF grants are not for ongoing maintenance, but for short and fixed projects.

Leave a comment

About TPF

The Perl Foundation - supporting the Perl community since 2000. Find out more at www.perlfoundation.org.

Recent Comments

  • szabgab: I understand people are worried about the size of the read more
  • Tom Hukins: DWIM Perl looks like an interesting idea. Whereas most grant read more
  • Nicholas Clark: I look at this grant and my first thought is read more
  • Leon Timmermans: My general impression is the same as David's. This is read more
  • David Golden: I think DWIM perl is a great idea and I read more
  • Smylers: Please support this. Dwim Perl is a very handy ‘everything read more

About this Entry

This page contains a single entry by Makoto Nozaki published on March 15, 2014 9:00 AM.

Grant Proposal: Perl::Lint - Yet Another Static Analyzer for Perl5 was the previous entry in this blog.

March 2014 Grant Proposals is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

OpenID accepted here Learn more about OpenID
Powered by Movable Type 4.38