2012Q1 Grant Proposal: Cooking Perl with Chef
Thu, 09-Feb-2012 by
Alberto Simões
edit post
<![CDATA[
<h1 id="Cooking-Perl-with-Chef">Cooking Perl with Chef</h1>
<dl>
<dt id="Name:">Name:</dt>
<dd>
<p>David Golden (DAGOLDEN/xdg)</p>
</dd>
<dt id="Email:">Email:</dt>
<dd>
<p>[Hidden]</p>
</dd>
<dt id="Amount-Requested:">Amount Requested:</dt>
<dd>
<p>USD $2000</p>
</dd>
</dl>
<h2 id="Synopsis">Synopsis</h2>
<p>I propose to create ready-to-use tools and supporting tutorials that teach Perl developers (and the sysadmins that support them) how to deploy Perl applications repeatable and reliably using the <a href="http://wiki.opscode.com/display/chef/Home">Chef</a> configuration management tool.</p>
<h2 id="Benefits-to-the-Perl-Community">Benefits to the Perl Community</h2>
<ul>
<li><p>First-class support for Perl in Chef, just like other web development languages</p>
</li>
<li><p>Open cookbooks and tutorials to encourage sharing techniques and practices for app deployment</p>
</li>
</ul>
<h2 id="Deliverables">Deliverables</h2>
<p><b>Code deliverables:</b></p>
<ul>
<li><p>Publish a Chef cookbook for Perl interpreter deployment</p>
</li>
<li><p>Publish a Chef cookbook for CPAN module deployment</p>
</li>
<li><p>Publish a Chef a cookbook for Plack application deployment</p>
</li>
<li><p>Improve the <a href="http://p3rl.org/Pantry">Pantry</a> command line tool for configuring servers with Chef Solo</p>
<ul>
<li><p>Add command-line configuration of "nodes" (servers) and "roles" (configuration patterns)</p>
</li>
<li><p>Add support for multiple environments (e.g. "dev", "test", "production") with different configuration options per environment</p>
</li>
</ul>
</li>
</ul>
<p><i>Note:</i> in Chef a "cookbook" encapsulates all code and metadata needed to automate configuration of particular resources. C.f. <a href="http://wiki.opscode.com/display/chef/Cookbooks">About Cookbooks (Opscode wiki)</a></p>
<p><b>Teaching deliverables:</b></p>
<ul>
<li><p>Publish a presentation-style introduction to using Perl with Chef</p>
</li>
<li><p>Publish a document-style "hello world" tutorial with step by step instructions for deploying a simple Plack app</p>
</li>
<li><p>Publish a document-style "real world" tutorial with step by step instructions for deploying a complete application stack for a real application</p>
</li>
<li><p>Release a video demonstration of "hello world" deployment, as either a screencast or a recording of a conference/seminar presentation</p>
</li>
</ul>
<h2 id="Project-Details">Project Details</h2>
<p><b>Motivation (aka "the problem")</b></p>
<p>Automated configuration management tools are increasingly part of the toolbox for Internet technology companies, but support for Perl and Perl application frameworks lags behind other dynamic languages. This makes Perl harder to integrate into an environment that already uses such tools and makes it harder for Perl-oriented companies to adopt such tools.</p>
<p>While these difficulties can be overcome, deploying Perl applications with configuration management tools remains an idiosyncratic black art, with the knowledge of how to do it locked up in the minds (or companies) that have done it. For someone new to configuration management, the learning curve is steep.</p>
<p><b>Approach (aka "the solution")</b></p>
<p>Over the last couple months, I have been experimenting with deploying Perl applications using the Chef configuration management tool. I demonstrated my "proof-of-concept" code at the Orlando Perl Workshop in January. I continue to tinker with this code and seek a grant to support and motivate turning this experiment into public resources for the Perl community at large.</p>
<p>Using configuration management tools for Perl applications requires addressing several issues that go beyond what typical Perl/CPAN toolchain programs deal with, such as:</p>
<ul>
<li><p>Idempotent deployment automation</p>
</li>
<li><p>Versioned library dependency management</p>
</li>
<li><p>Application restart when configuration (including dependencies) changes</p>
</li>
<li><p>Multiple perls and multiple apps deployed simultaneously on a single server</p>
</li>
</ul>
<p>My proof-of-concept combines existing tools like perlbrew, local::lib, carton, Plack, and so on to achieve these goals. While the proof-of-concept tools work, they need further refinement, documentation end testing before they are release-ready for general use.</p>
<p>Once these cookbooks and tools are created, I want to make them easy to learn. Since not everyone learns the RTFM way, I propose to create two step-by-step tutorials that walk someone though deploying an application from bare (virtual) metal to live server.</p>
<p>The first tutorial will be the configuration management equivalent of "hello world" -- it will be the smallest code that can show the essential components and steps of the solution. Since not everyone learns from reading tutorials and since there appears to be popular appeal for screencast tutorials, I will also record a video of this tutorial. (Or, if an appropriate conference/seminar opportunity arises to make such a demonstration during the grant period, I will provide that instead.)</p>
<p>The second tutorial will be a "real world" one that shows how a full application -- including supporting, non-Perl components like a web-server, database, caching layer, etc. -- can be deployed and configured. I have not settled on an application for this tutorial, but I consider the CPAN Testers Metabase or the <a href="http://jitterbug.pl">Jitterbug</a> continuous integration server to be leading candidates.</p>
<h2 id="Inch-stones">Inch-stones</h2>
<p>See "Deliverables" for general work-blocks.</p>
<p>Each of the code deliverables will be broken into "coding", "documentation" and "release" inch-stones.</p>
<p>Written teaching documents will be broken into "rough draft", "peer review", "final draft" and "release" inch-stones.</p>
<p>The video deliverable will be broken into "storyboard", "recording", "post-production" and "release" inch-stones.</p>
<h2 id="Project-Schedule">Project Schedule</h2>
<p>The project is estimated to take three months. I will begin immediately upon being awarded the grant.</p>
<h2 id="Completeness-Criteria">Completeness Criteria</h2>
<p>All code and teaching deliverables will be published online under open licenses.</p>
<ul>
<li><p>Cookbooks will be published to the <a href="http://community.opscode.com/">Chef Community Cookbooks</a> site (which is like CPAN for Chef).</p>
</li>
<li><p>Pantry will be published to CPAN.</p>
</li>
<li><p>Tutorials will be published either to my blog or to an appropriate Perl tutorials site.</p>
</li>
<li><p>Video will be published to YouTube and/or Perl-specific video sites.</p>
</li>
</ul>
<p>Code, presentations and documents will also be made available via Github for community access and improvement.</p>
<h2 id="Bio">Bio</h2>
<p>I have a long history contributing to the Perl 5 community:</p>
<ul>
<li><p>Active Perl 5 Porter responsible for two dev releases, several new features and the occasional bug fix.</p>
</li>
<li><p>Developer of the first several releases of <a href="http://www.strawberryperl.com">Strawberry Perl</a>.</p>
</li>
<li><p>Prolific <a href="https://metacpan.org/author/DAGOLDEN/">CPAN author</a>. Contributor to or co-maintainer of a large subset of the CPAN toolchain.</p>
</li>
<li><p>Leader of the <a href="http://p3rl.org/CPAN::Meta::Spec">CPAN Meta Spec</a> v2 <a href="http://www.dagolden.com/index.php/759/cpan-meta-spec-version-2-released/">revision process</a></p>
</li>
<li><p>Lead architect and manager of the <a href="http://metabase.cpantesters.org/">CPAN Testers Metabase</a> that collects nearly a million CPAN Testers reports every month.</p>
</li>
<li><p>Moderately-frequent <a href="http://www.dagolden.com/">blogger</a> and conference/seminar <a href="http://www.dagolden.com/index.php/talks/">presenter</a> on Perl-related topics.</p>
</li>
</ul>
]]>
Comments (12)
I like that this builds on an already successful project (Chef). David has a good track record of getting things done, which is a big plus.
Does this need any buy-in from the Chef developers, or can it stand on its own?
Also, why Chef and not Puppet or some other tool? Is there any agreement in the Perl community as to what the best choice is? FWIW, we've started using Puppet at my $DAY_JOB.
Overall, I think this looks like a useful project.
Definitely seems useful. Too often we in Perland are in a bubble and don't participate fully with outside non-perl projects.
My only nibble is like autarch, I wonder why Chef and not Puppet. In my experience Puppet is more popular and usage is on the rise.
My experience has been the opposite. Chef is more popular among the developers I've run into.
I'm hoping this happens. I'm just learning Chef and trying to automate the configuration of our Perl servers. I was probably going to end up writing some of this myself.
Chef is a Ruby-based tool.
Ruby breaks backward compatibility too often even with minor language version upgrade.
So, It's harder to maintain working language version across servers than do real sysadmin tasks.
I can't understand who use such unstable thing for tasks requiring robustness.
I think it's better to raise fund for Rex( http://rexify.org/ ) instead of Chef.
It's Perl's counterpart to Chef.
I have absolutely zero doubt that David will deliver what he promises. His track record in following through on what he starts is excellent.
As for the choice of tool, I'd also prefer to see puppet used as well, simply because $work uses it heavily. That being said, I believe that David's work on Chef would be valuable, not only standing on its own, but also as research and example on how other volunteers can build similar setups with other tools. His inclusion of documentation and learning material in the grant may go a long way towards sharing the design work.
I left a comment before, but it seems lost in the moderation queue, so I'll try again.
Re: Chef developer support -- no direct support needed. Their community cookbook site seems open to just about any reasonable contribution.
Re: Chef vs puppet -- I think this is like emacs-vs-vi. I looked at both and Chef fit my brain better. Support for standalone mode with Chef Solo also seems better than the equivalent with Puppet, which has nice properties that I won't go into here, but might blog about someday.
As Steffen said, I think the techniques/patterns I'm developing for Chef could be translated to Puppet by a reasonably skilled Puppet hacker.
I'd like to see better integration for Perl projects in such projects. Companies often want use the same tools for all applications they develop (no matter what language the application is written in). So I support this grant application!
I think it is very important to have such integration for Perl with multiple tools and I have seen what David is capable of so I am fully supporting this request.
IMHO integrating with Chef is a good first step but I'd like to see plans, e.g. a follow-up grant, either from David or from someone else, to create the integration with Puppet and Rex as well.
I am all for this. ... Assuming that some parts are built in a platform agnostic way.
What I would really love is a good interface to apt. I have tried to write a script to run immediately after installing a fresh Ubuntu. I finally gave up because I couldn't get useful information about what was installed, installable, which repositories were available, etc etc. That would be a great thing. Once that was running, I would be happy to incorporate it into Alien::Base or support its existence in some other installer mechanism.
Shoot, the above was meant for the CPAN::non post. Too many tabs fail I think. Sorry about that.
A huge +1 to this grant. It will be extremely useful in many contexts.
