2010Q3 Grant Proposal: CPAN to deb autopackaging
Sun, 01-Aug-2010 by
Alberto Simões
edit post
<![CDATA[
<h1>CPAN to deb autopackaging</h1>
<dl>
<dt><strong>Name:</strong></dt>
<dd>
<p>Jozef Kutej</p>
</dd>
<dt><strong>Email:</strong></dt>
<dd>
<p>[hidden email]</p>
</dd>
<dt><strong>Amount Requested:</strong></dt>
<dd>
<p>$3000</p>
</dd>
</dl>
<p>
</p>
<h2>Synopsis</h2>
<p>The aim of the project would be to create automatic CPAN distribution to
Debian deb packaging system. Then use it to autopackage and backport at
least 50% of the CPAN distributions resulting in a Debian repository with
multiple thousand Perl related packages.</p>
<h2>Benefits to the Perl Community</h2>
<p>Debian+Ubuntu operating systems are very popular among Perl developers.
According to a <a href="http://perlide.org/poll201004/">http://perlide.org/poll201004/</a> - used by %37 percent.
The Debian packaging proved over the time to be a consistent and a clean
way to install software to the OS. Having CPAN distributions packaged as
Debian packages would create a way to cleanly install, reinstall and
uninstall Perl modules. Taking care of file conflicts and the records of
which files comes from which source improving the deployment quality and
maintainability.</p>
<p>One quite important advantage of another layer above the CPAN is the
ability to apply custom patches in a well controlled and defined way.</p>
<p>
</p>
<h2>Deliverables</h2>
<ul>
<li>
<p>Finish <code>dh-make-pm</code> (<a href="/Debian/Apt/PM.html">the Debian::Apt::PM manpage</a>) and package at least 50% (10k) of CPAN distributions.</p>
</li>
<li>
<p>Create and maintain Debian repository for this packages + provide hosting
for it for at least 1 year.</p>
</li>
</ul>
<p>
</p>
<h2>Project Details</h2>
<dl>
<dt><strong>Overview</strong></dt>
<dd>
<p>I'm already working on and experimenting with automated Debian packaging
for couple of months already. The task is not easy. The biggest problem
of CPAN is that there is no way how to declare non-CPAN dependencies like
shared libraries, so it has to be done manually. Another problem is the CPAN
module version to CPAN distribution version (that mostly match but not always)
and then the Debian package to CPAN distribution mapping. This is needed
to declare minimum package version when setting up dependencies. Another
minor problem is that not all requires and build_requires are listed, which
is not so common thanks to CPAN testers. Sometimes the CPAN distribution
are not libraries but applications. And sometimes the distributions lack
the META.yml file that makes the packager life harder.</p>
<p>I would like to build the automates system on top of the work of Debian-Perl
group. The most of the most important CPAN distributions are already
packaged and needs to be just backported for the stable distribution.</p>
<p>Just note: Autopackaged ≠maintained.</p>
</dd>
<dt><strong>Isn't this a task for Debian developers?</strong></dt>
<dd>
<p>Yes and no.</p>
<p>Currently Debian squeezy (testing) has 2000+ Perl related packages. That
is ~10% of all CPAN distributions. The numbers, both CPAN and Debian are
growing. But for Debian the amount is limited as the packages index files
needs to be loaded and processed on different machines with often limited
resources. In this regard it makes sense to create a special repository
dedicated just for Perl.</p>
<p>Second reason why no, is that the Debian developers work on testing/unstable
release. Stable is stable thanks to not upgrading, just doing security bug
fixing. For a Perl world having 2-3 years old CPAN distributions can be
acceptable in some cases but for most cases when a Perl product is actively
developed no.</p>
</dd>
<dt><strong>Why bother with Debian packaging?</strong></dt>
<dd>
<p>As I mentioned earlier another layer above CPAN allows more flexibility
that Linux distribution maintainers need in order to work with "foreign"
software. That means adding custom patches, tweaking and customizing.
Distribution packaging also keed track of the installed files and of
course the package names and versions. Installing, upgrading, downgrading, removing is a
part of the system. It is also easy to find from which package the
installed file is coming from. Or lookup a specific file in a package
indexes. Packaging also makes sure that no two packages overwrites the
same file.</p>
</dd>
<dt><strong>How will it work? Or how does it work so far?</strong></dt>
<dd>
<p><code>dpkg-scanpmpackages</code> can index Perl modules in a Debian repository.
<code>apt-pm</code> or <a href="/Debian/Apt/PM.html">the Debian::Apt::PM manpage</a> can use them to look up minimal package
version for a given required module version. <code>dh-make-pm</code> will build
a package and recursively all the packages that are needed for it.
Ex. - <code>dh-make-pm --cpan Geo::KML</code>. If also apply local patches using <a href="/CPAN/Patches.html">the CPAN::Patches manpage</a>.
Debian-Perl group patches and dependencies are extracted using <code>cpan-patches-update-from-debian</code>.</p>
<p>Current status of setting up the build system is here:</p>
<p><a href="http://github.com/jozef/Debian-Apt-PM/blob/master/lib/Debian/Apt/PM/SettingUpBuildSystem.pod">http://github.com/jozef/Debian-Apt-PM/blob/master/lib/Debian/Apt/PM/SettingUpBuildSystem.pod</a></p>
</dd>
</dl>
<p>
</p>
<h2>Inch-stones</h2>
<ul>
<li>
<p>Backport 1600+ modules that are maintained by Debian-Perl group for
Debian Lenny (stable)</p>
<p>Note my plan is to have 1:1 relation CPAN distribution to Debian package,
while original Debian has a couple of bundles to minimize the repository
packages index sizes.</p>
</li>
<li>
<p>Continue with some CPAN author sample (ex.
<a href="http://jozef.kutej.net/2010/07/cpan-authors-sample.html">http://jozef.kutej.net/2010/07/cpan-authors-sample.html</a>) and package
their modules.</p>
</li>
<li>
<p>try to go beyond 10k packages</p>
</li>
<li>
<p>Set-up repository that can handle so many Debian packages</p>
</li>
<li>
<p>Create a repository web site with a detailed how-to reproduce so that
anyone can easily build Debian packages for his/her internal purposes.</p>
</li>
<li>
<p>File bugs when build dependencies are missing.</p>
</li>
<li>
<p>Improve <a href="/CPAN/Patches.html">the CPAN::Patches manpage</a> example set by collecting patches from RT to
allow some unmaintained but still useful modules pass their tests and be
packaged.</p>
</li>
</ul>
<p>
</p>
<h2>Project Schedule</h2>
<p>The project should be finish by the end of this year. Having funding from
TPF will allow me to work starting from October 10-20h a week on it.</p>
<p>
</p>
<h2>Completeness Criteria</h2>
<p>The grant can be simply evaluated by counting the number of packages
in a repository. Installing them to a Debian Lenny
machine.</p>
<p>
</p>
<h2>Bio</h2>
<ul>
<li>
<p>JAPH ;-)</p>
</li>
<li>
<p><a href="http://search.cpan.org/~jkutej/">http://search.cpan.org/~jkutej/</a></p>
</li>
<li>
<p><a href="http://use.perl.org/journal.pl?op=list&uid=8299">http://use.perl.org/journal.pl?op=list&uid=8299</a></p>
</li>
<li>
<p><a href="http://jozef.kutej.net/">http://jozef.kutej.net/</a></p>
</li>
</ul>
]]>
Comments (8)
I just gotta say: I tried to do this for Fink, which is a DPKG source-based distro for Mac OS X. It was *hard* and I got rapidly buried in the special cases.
If Jozef can pull this off, it would be cool and I would be very impressed, but my pessimistic side predicts failure.
This is probably my favorite proposal from this list. This has been on my tuits list for ages, it would make my programming life a lot easier, not to mention.
Some questions I do have: What platforms are you planning to support? x86 + x64 or more?
What versions of Debian and Ubuntu are you planning to support for your repository?
Yes it will be not easy and it will be not perfect (perfect are the official Debian packages), it will need a lot of patience, but I'm convinced that it is possible.
My plan is only i386 and amd64 as I have no access to any other platforms. But there is no problem for anyone who has a different platform to port it further. It will be an automated way that anyone can use at his company or for his project and build a packages with a repository for his own project. The final public CPAN-deb repository will be a proof of concept. As of the distribution, I plan to work with Debian stable. Lenny for the moment, by the end of the year probably Squeeze.
disclaimer: i'm a maintainer of 700+ perl module packages for mandriva.
creating the packages is not that big a deal with a good cpanplus backend. but creating the packages is only half the problem.
the real problem is maintaining them - and the grant only says that they will be maintained for 1 year - what happens afterwards? what are your plans? will you re-apply for another maintenance grant? will your work / infrastructure / whatever will be available for volunteers to maintain it?
if you stop your maintenance work after 1 year, then this grant is moot. it will do more harm than good, leaving your users in the cold.
not big deal? couple of people already tried and no distribution so far packaged a significant part of the CPAN. the basic tools works as long as the CPAN packages are done properly and are pure Perl. with the rest we have a problem. if there is no additional system, then we have a problem every time a new version comes out.
naturally the project will be 100% open. code, documentation and infrastructure to anyone wanting to support it. that is why I apply for a grant at Perl Foundation. if I would like to do something closed then I'll search for a business partners.
the first year will be a test run. burning money for at least two servers. one with the repository of ~2.5GB of data + bandwidth. The other one full time busy with building and indexing packages. what will be after one year? I believe that the Debian community is strong enough and would be interested in the project enough to find volunteers with some spare time, cpu and bandwidth to support. if not, it will deserve to die and we can move on to something else with the experience that was gained in that year.
to not be so Debian-centric, there is something for everyone. look at http://github.com/jozef/CPAN-Patches-Debian-Set where is the set of Debian package information extracted. this is patches and dependencies. anyone can easily look and see what patches/dependencies Debian has and fix some CPAN distribution with them. this is one part of the automated system. it is build-up on top of the hard work of Debian-Perl group, not trying to restart from scratch.
My disclaimer: I'm a Fedora maintainer and the author of cpanspec, the tool (I think) most Fedora maintainers are using to build rpms from CPAN packages.
I've thought any number of times about building a as-much-of-CPAN-as-possible repo using cpanspec, but only as a way of stress-testing cpanspec. I wouldn't want anyone to *use* those packages, since they wouldn't be tested, maintained, etc.
I guess I just don't see the point...
Would you please also make sure http://debian.pkgs.cpan.org/debian/ which is OLD! and OUT OF DATE! and obviously a previous attempt of the same, is removed? Or can anyone do that? I tried to mail the maintainer but did not receive any reaction.
