2010Q3 Grant Proposal: CPAN to deb autopackaging

8 Comments

CPAN to deb autopackaging

Name:

Jozef Kutej

Email:

[hidden email]

Amount Requested:

$3000

Synopsis

The aim of the project would be to create automatic CPAN distribution to Debian deb packaging system. Then use it to autopackage and backport at least 50% of the CPAN distributions resulting in a Debian repository with multiple thousand Perl related packages.

Benefits to the Perl Community

Debian+Ubuntu operating systems are very popular among Perl developers. According to a http://perlide.org/poll201004/ - used by %37 percent. The Debian packaging proved over the time to be a consistent and a clean way to install software to the OS. Having CPAN distributions packaged as Debian packages would create a way to cleanly install, reinstall and uninstall Perl modules. Taking care of file conflicts and the records of which files comes from which source improving the deployment quality and maintainability.

One quite important advantage of another layer above the CPAN is the ability to apply custom patches in a well controlled and defined way.

Deliverables

  • Finish dh-make-pm (the Debian::Apt::PM manpage) and package at least 50% (10k) of CPAN distributions.

  • Create and maintain Debian repository for this packages + provide hosting for it for at least 1 year.

Project Details

Overview

I'm already working on and experimenting with automated Debian packaging for couple of months already. The task is not easy. The biggest problem of CPAN is that there is no way how to declare non-CPAN dependencies like shared libraries, so it has to be done manually. Another problem is the CPAN module version to CPAN distribution version (that mostly match but not always) and then the Debian package to CPAN distribution mapping. This is needed to declare minimum package version when setting up dependencies. Another minor problem is that not all requires and build_requires are listed, which is not so common thanks to CPAN testers. Sometimes the CPAN distribution are not libraries but applications. And sometimes the distributions lack the META.yml file that makes the packager life harder.

I would like to build the automates system on top of the work of Debian-Perl group. The most of the most important CPAN distributions are already packaged and needs to be just backported for the stable distribution.

Just note: Autopackaged ≠ maintained.

Isn't this a task for Debian developers?

Yes and no.

Currently Debian squeezy (testing) has 2000+ Perl related packages. That is ~10% of all CPAN distributions. The numbers, both CPAN and Debian are growing. But for Debian the amount is limited as the packages index files needs to be loaded and processed on different machines with often limited resources. In this regard it makes sense to create a special repository dedicated just for Perl.

Second reason why no, is that the Debian developers work on testing/unstable release. Stable is stable thanks to not upgrading, just doing security bug fixing. For a Perl world having 2-3 years old CPAN distributions can be acceptable in some cases but for most cases when a Perl product is actively developed no.

Why bother with Debian packaging?

As I mentioned earlier another layer above CPAN allows more flexibility that Linux distribution maintainers need in order to work with "foreign" software. That means adding custom patches, tweaking and customizing. Distribution packaging also keed track of the installed files and of course the package names and versions. Installing, upgrading, downgrading, removing is a part of the system. It is also easy to find from which package the installed file is coming from. Or lookup a specific file in a package indexes. Packaging also makes sure that no two packages overwrites the same file.

How will it work? Or how does it work so far?

dpkg-scanpmpackages can index Perl modules in a Debian repository. apt-pm or the Debian::Apt::PM manpage can use them to look up minimal package version for a given required module version. dh-make-pm will build a package and recursively all the packages that are needed for it. Ex. - dh-make-pm --cpan Geo::KML. If also apply local patches using the CPAN::Patches manpage. Debian-Perl group patches and dependencies are extracted using cpan-patches-update-from-debian.

Current status of setting up the build system is here:

http://github.com/jozef/Debian-Apt-PM/blob/master/lib/Debian/Apt/PM/SettingUpBuildSystem.pod

Inch-stones

  • Backport 1600+ modules that are maintained by Debian-Perl group for Debian Lenny (stable)

    Note my plan is to have 1:1 relation CPAN distribution to Debian package, while original Debian has a couple of bundles to minimize the repository packages index sizes.

  • Continue with some CPAN author sample (ex. http://jozef.kutej.net/2010/07/cpan-authors-sample.html) and package their modules.

  • try to go beyond 10k packages

  • Set-up repository that can handle so many Debian packages

  • Create a repository web site with a detailed how-to reproduce so that anyone can easily build Debian packages for his/her internal purposes.

  • File bugs when build dependencies are missing.

  • Improve the CPAN::Patches manpage example set by collecting patches from RT to allow some unmaintained but still useful modules pass their tests and be packaged.

Project Schedule

The project should be finish by the end of this year. Having funding from TPF will allow me to work starting from October 10-20h a week on it.

Completeness Criteria

The grant can be simply evaluated by counting the number of packages in a repository. Installing them to a Debian Lenny machine.

Bio

8 Comments

I just gotta say: I tried to do this for Fink, which is a DPKG source-based distro for Mac OS X. It was *hard* and I got rapidly buried in the special cases.

If Jozef can pull this off, it would be cool and I would be very impressed, but my pessimistic side predicts failure.

This is probably my favorite proposal from this list. This has been on my tuits list for ages, it would make my programming life a lot easier, not to mention.

Some questions I do have: What platforms are you planning to support? x86 + x64 or more?

What versions of Debian and Ubuntu are you planning to support for your repository?

Yes it will be not easy and it will be not perfect (perfect are the official Debian packages), it will need a lot of patience, but I'm convinced that it is possible.

My plan is only i386 and amd64 as I have no access to any other platforms. But there is no problem for anyone who has a different platform to port it further. It will be an automated way that anyone can use at his company or for his project and build a packages with a repository for his own project. The final public CPAN-deb repository will be a proof of concept. As of the distribution, I plan to work with Debian stable. Lenny for the moment, by the end of the year probably Squeeze.

disclaimer: i'm a maintainer of 700+ perl module packages for mandriva.

creating the packages is not that big a deal with a good cpanplus backend. but creating the packages is only half the problem.

the real problem is maintaining them - and the grant only says that they will be maintained for 1 year - what happens afterwards? what are your plans? will you re-apply for another maintenance grant? will your work / infrastructure / whatever will be available for volunteers to maintain it?

if you stop your maintenance work after 1 year, then this grant is moot. it will do more harm than good, leaving your users in the cold.

not big deal? couple of people already tried and no distribution so far packaged a significant part of the CPAN. the basic tools works as long as the CPAN packages are done properly and are pure Perl. with the rest we have a problem. if there is no additional system, then we have a problem every time a new version comes out.

naturally the project will be 100% open. code, documentation and infrastructure to anyone wanting to support it. that is why I apply for a grant at Perl Foundation. if I would like to do something closed then I'll search for a business partners.

the first year will be a test run. burning money for at least two servers. one with the repository of ~2.5GB of data + bandwidth. The other one full time busy with building and indexing packages. what will be after one year? I believe that the Debian community is strong enough and would be interested in the project enough to find volunteers with some spare time, cpu and bandwidth to support. if not, it will deserve to die and we can move on to something else with the experience that was gained in that year.

to not be so Debian-centric, there is something for everyone. look at http://github.com/jozef/CPAN-Patches-Debian-Set where is the set of Debian package information extracted. this is patches and dependencies. anyone can easily look and see what patches/dependencies Debian has and fix some CPAN distribution with them. this is one part of the automated system. it is build-up on top of the hard work of Debian-Perl group, not trying to restart from scratch.

My disclaimer: I'm a Fedora maintainer and the author of cpanspec, the tool (I think) most Fedora maintainers are using to build rpms from CPAN packages.

I've thought any number of times about building a as-much-of-CPAN-as-possible repo using cpanspec, but only as a way of stress-testing cpanspec. I wouldn't want anyone to *use* those packages, since they wouldn't be tested, maintained, etc.

I guess I just don't see the point...

Would you please also make sure http://debian.pkgs.cpan.org/debian/ which is OLD! and OUT OF DATE! and obviously a previous attempt of the same, is removed? Or can anyone do that? I tried to mail the maintainer but did not receive any reaction.

About TPF

The Perl Foundation - supporting the Perl community since 2000. Find out more at www.perlfoundation.org.

Recent Comments

  • Michiel Beijen: Would you please also make sure http://debian.pkgs.cpan.org/debian/ which is OLD! read more
  • Steven Pritchard: My disclaimer: I'm a Fedora maintainer and the author of read more
  • Jozef: not big deal? couple of people already tried and no read more
  • jerome quelin: disclaimer: i'm a maintainer of 700+ perl module packages for read more
  • Jozef: My plan is only i386 and amd64 as I have read more
  • Jozef: Yes it will be not easy and it will be read more
  • Leon Timmermans: This is probably my favorite proposal from this list. This read more
  • Chris Dolan: I just gotta say: I tried to do this for read more

About this Entry

This page contains a single entry by Alberto Simões published on August 1, 2010 3:07 PM.

2010Q3 Grant Proposal: Perlbal documentation was the previous entry in this blog.

2010Q3 Grant Proposal: Improve Parrot Embed/Extend Subsystem Tests and Documentation is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

OpenID accepted here Learn more about OpenID
Powered by Movable Type 4.38