2012Q1 Grant Proposal: Alien::Base - Base classes for Alien:: modules

14 Comments
Name:

Joel Berger

Email:

[hidden email]

Amount Requested:

$500

Synopsis

Artur Bergman's documentation for the Alien namespace specifically says that there is no framework for Alien:: modules, however, in writing the Alien::GSL manpage I find that most of the code I am writing has nothing to do with the GSL library and could as easily apply to many external libraries. I intend for Alien::Base to comprise some base classes for other Alien:: modules, including Alien::GSL.

Benefits to the Perl Community

Perl is great, CPAN is great, CPAN.pm and cpanminus are great, but if a module depends on an external library that is not installed, none of these can help. The Alien namespace has attempted to meet this demand by providing installers for external libraries needed by CPAN modules.

Sadly, comparitively few libraries are provided. In my opinion this is because writing the Alien module itself is too hard. With an Alien::Base system in place, perhaps more libraries could be provided and Perl would have an even larger base of available code to build from.

Deliverables

A CPAN distribution named Alien::Base, comprised of at least the classes:

  • Alien::Base - used as the base class for an Alien module (e.g. Alien::OtherLibrary), which is called by the dependent module (e.g. Something::Other::XS).

  • Alien::Base::ModuleBuild - a the Module::Build manpage subclass which handles the actions of fetching, building and installing the library.

  • Tests

  • Documentation

Project Details

Alien::Base::ModuleBuild will be able to

  • Query an FTP or HTTP server for a list of files

  • Fetch a file (archive) from either type of server, based on a required or else newest version

  • Extract the archive to a temporary directory

  • Perform a series of system calls to configure, build and install the library (to a the File::ShareDir manpage type folder)

  • Extract some information about the library and store it in a the Module::Build::ConfigData manpage storage module

Alien::Base will be able to

  • Provide the information about the library's local installation

  • Make the library available to a dependent module and/or its builder

The aim will be to allow much, if not all, of the subclass (i.e. Alien::OtherLibrary) specifics to be set with parameters/properties just like the Module::Build manpage installations, rather than by needing to write extra methods to do any of the above-listed tasks. I envision that the Build.PL for Alien::GSL could look like this (properties like alien_ are from Alien::Base):

 use strict;
 use warnings;
 use Alien::Base::ModuleBuild;
 my $builder = Alien::Base::ModuleBuild->new(
   module_name => 'Alien::GSL',
   dist_abstract => 'Easy installation of the GNU Scientific Library',
   license => 'perl',
   configure_requires => {
     'Alien::Base' => 0,
   },
   requires => {
     'perl' => '5.8.1',
     'Alien::Base' => 0,
   },
   dist_author => 'Joel A. Berger <[email protected]>',
   alien_name => 'gsl',
   alien_source_ftp => { 
     server  => 'ftp.gnu.org',
     folder  => '/gnu/gsl',
     pattern => qr/^gsl-([\d\.])+\.tar\.gz$/,
   },
 );
 $builder->create_build_script;

Initially I intend to focus on GNU libraries as they are fairly consistent in their build process. This should allow Alien:: authors to wrap this large and important set of libraries, employing only minimal configuration. I intend to use this for the Alien::GSL manpage and perhaps something like Alien::NCurses (i.e. a library I am not as familiar with) if I feel so inclined.

Hopefully the process will be sufficiently configurable so as to allow other non-GNU libraries to be wrapped easily. Certainly something involved like the Alien::SDL manpage isn't a prime candidate for this framework, but there are many smaller libraries which might benefit from being wrapped in this manner.

After initial releases, I hope that the community will help to make Alien::Base even more broadly applicable to more libraries.

Inch-stones

Each of the bullets under Project Details will serve as inch-stones.

Project Schedule

I have already begun working on Alien::Base; see it at https://github.com/jberger/Alien-Base. Development shouldn't take too long; I imagine only a few total weeks work should do the trick. Unfortunately my time cannot be given completely to this task since I am also currently working on my thesis (see Bio). Therefore, I propose that my project be 3-4 months (calendar) duration. Hopefully I will not need all that time. Also I have been trying to include tests as I code (countering a usual fault of mine) which is taking longer. This grant will help me to focus on this project for a shorter burst of development to completion, rather than working on it here and there as able.

Completeness Criteria

Distribution released to CPAN, complete with a decent test suite and documentation. Further convert the Alien::GSL manpage to the Alien::Base framework, giving an in-the-wild example for others to follow.

Bio

I am a Physics Ph.D. candidate at the University of Illinois at Chicago. I am writing a large simulation, a major part of my thesis, which uses Perl and my the Math::GSLx::ODEIV2 manpage module. This in turn requires version 1.15 of the GNU Scientific Library (GSL). I want my committee to be able to try out the simulation on their own; I have worked hard to make sure that it has a nice API, but certainly I cannot install it for each of them on their computers. Unfortunately, very few Linux distros come with this newest version of GSL (although this is always improving) and Windows users certainly will not have it.

My goal is to be able to have my committee members install Perl (Strawberry presumably on Windows) and my simulation module, which depends on the Math::GSLx::ODEIV2 manpage and therefore the Alien::GSL manpage (which would now depend on the Alien::Base manpage). From their perspective, I want it to be a totally transparent installation.

Since I have already made this effort, I would like it if my work on Alien::GSL could be used to make the task of writing other Alien:: modules easier for other authors.

14 Comments

Please note that the POD renderer here is changing the L links to "the Module::Name manpage" and breaking the links. I have reposted to a GH Gist for those who would prefer: https://gist.github.com/1616923

I wasn't going to comment on that, but now that you have asked. Of course this may be biased.

I think that that mechanism has a distinct flaw in that it depends on there being hooks for each package management system. Further this can only be a Linux solution (and as I said, not even fully cross-linux let alone cross platform). I can't imagine a module author who would add such a system as a dependency which cannot be satisfied on a restrict set of platforms.

I think that if such a mechanism were employed it should be a single use-case inside Alien::Base. That said, when I started the Alien::Base project, I considered all kinds of mechanisms, and I decided that for simplicity and broadest application, I should pick one installation process, namely building from source.

Alien::Base short-circuits if it finds a system installed library, otherwise it builds the library in a File::ShareDir location. It will have hooks to allow downloading pre-compiled packages, in principle it would not be hard to add linux-based package management hooks too. With this system in place module authors can be more confident that the library in question can be installed everywhere.

I want to clarify one thing from my previous comment:

Alien::Base will short-circuit if it finds a system-wide installation of a library. If that is not found then no change is made to the system outside of the File::ShareDir location. This was a design choice I made to because:

1) it is a path of least surprise when installing a Perl module (doesn't affect the system)
2) it is safest
3) doesn't require root privileges (leads to:)
4) works with local::lib/perlbrew

A build from source would be preferred, but of course some Perl (especially Windows) cannot build modules. For this purpose, pre-compiled downloads are possible, though they will still install inside a File::ShareDir location. If I understand PPM, this should be easy to package for AS-Perl.

All that said, hooking in to the platform's package manager to install dependencies must be root and cannot be scoped using File::ShareDir nor be localized with local::lib/perlbrew. If the community really wants this facility it could be added, but I don't have it on my todo list. And again, since there is no standard package manager, this mechanism is neither cross-platform nor even cross-linux.

I sketched out some rough ideas for this sort of thing years ago, but it never got traction. It might be useful to you now as you consider your work: Portable Alien Library System

I'm going to say the same thing for all the grant requests this round ...

As a general rule, I think grants should be used to fund development on projects that are already established.

Speculative projects are just that, speculative. Even if the grant is completed, it's really hard to know whether the resulting work will be useful to anyone.

When funding work on an existing project, we already have some idea of whether or not that project has traction in the wider Perl community.

Perhaps I shouldn't say this, I tend to agree with you actually. Given last quarter's submissions however, I thought that this work was at least worth submitting.

I do know that the PDL Porters are anxiously awaiting some breakthrough in the Alien:: world since the major hinderance to PDL adoption seems to be external dependencies.

(Previous comment (PDL Alien::) was me)

I see this project a little differently, because there are already many Alien modules, many of which are reinventing the wheel. Joel is proposing to extract the reusable bits from what he is already working on, which makes it not really greenfield ("speculative") work.

As I pondered this same topic years ago and never had the tuits, I'm supportive of someone taking the initiative now to define some standard tools for Alien::* packages.

I have, at various times in the past, worked on other people's Alien::-style distributions, written my own (Alien::ROOT must clock in as the single largest install footprint of any Perl module), and written other XS modules that inline some of the typical Alien:: functionality of auto-fetching or building external libraries.

Like David Golden, I've thought many times about abstracting some of the tools into a reusable library. My excuse is that I always found an itchier itch to scratch.

At the proposed expense for the grant, I believe such a library would provide tangible benefit to module authors and it would be a steal.

My only concern is that the proposed library sounds like it may make too many assumptions about the library being built. For that reason, I very much hope that each of the build steps will be overridable/hookable (obtain, extract, configure, build, test, install, package from the top of my head).

+1

My only concern is that the proposed library sounds like it may make too many assumptions about the library being built. For that reason, I very much hope that each of the build steps will be overridable/hookable (obtain, extract, configure, build, test, install, package from the top of my head).

For now, the build process is going to be an array of strings, with some placeholders (i.e. target folder). The placeholders are substituted and then strings are executed via `system`. As if this moment the default is

[ '%pconfigure --prefix=%s', 'make', 'make install' ]
where %p is './' on linux and %s is the File::ShareDir source.

In principal there is no reason that if the array element were a CodeRef that it could be run rather than system('string'). In fact, I think I might add this to the roadmap. Still that takes it away from being "config only", but not too far :)

Of course that is the build process. For the obtain extract etc, I am trying to be extensible and provide a sub-classable api. Remember that the whole system is a subclass of Module::Build and I hope to present it that way. Its not ready yet, but its the goal.

Speaking as a PDL porter, I can say that having a useful starting base for Alien modules would be a huge help. We have a number of libraries that appear to have interest primarily among PDL users and developers, PLplot and Slatec being big ones, and GSL being another. Joel has produced Alien::GSL, but the other two remain open problems and will likely require multiple Alien modules to fix (PLplot requires CMake; Slatec requires a Fortran compiler). Having a base module and a cookbook with ways of wrapping up libraries or compilable binaries would be an enormous help here, and this is the hole that I imagine Alien::Base would fill.

There is, of course, the related proposal about creating a library that knows how to talk with the different Linux packagers. I think that is also a wonderful idea, but should ultimately be folded into the Alien concept. It also does not provide any help to Windows or Mac users, which are two large target audiences of PDL. The PDL Pumpking, Chris Marshall, has worked very hard to make PDL more and more cross-platform with each new release, and I think that we would prefer to see a solution that we can target for all major platforms rather than just Linux.

So, I would endorse this project, contingent upon it having lots of documentation that illustrates how to customize the system and install both executables and libraries.

(Full disclosure: I know Joel from the PDL mailing list, more recently from meeting in person at Chicago.pm, and also from many discussions on #pdl at irc.perl.org.)

PE File is the executable file type in the windows OS, we also know it as the exe, dll and sys file extension. The PEFile tool can be use to get information from the files like pe header structure and Stream (ADS) Information, but not only that, this application can give you a lot more information on a EXE file and it can output the information to HTML file

Leave a comment

About TPF

The Perl Foundation - supporting the Perl community since 2000. Find out more at www.perlfoundation.org.

Recent Comments

  • Audrey Mascroft: PE File is the executable file type in the windows read more
  • David Mertens: Speaking as a PDL porter, I can say that having read more
  • Joel Berger: Of course that is the build process. For the obtain read more
  • Joel Berger: My only concern is that the proposed library sounds like read more
  • Steffen Mueller: I have, at various times in the past, worked on read more
  • dagolden.com: I see this project a little differently, because there are read more
  • Joel Berger: (Previous comment (PDL Alien::) was me) read more
  • Anonymous: Perhaps I shouldn't say this, I tend to agree with read more
  • autarch.urth.org: I'm going to say the same thing for all the read more
  • dagolden.com: I sketched out some rough ideas for this sort of read more

About this Entry

This page contains a single entry by Alberto Simões published on February 4, 2012 1:42 PM.

2012Q1 Grant Proposal: EPublisher Website was the previous entry in this blog.

2012Q1 Grant Proposal: Cooking Perl with Chef is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

OpenID accepted here Learn more about OpenID
Powered by Movable Type 4.38