We have received the following grant application "Test::Simple + Test::Stream stabilization and merge into perl-blead". Please leave feedback in the comments field by November 27th, 2015. If your comment does not appear in 24 hours, contact me at tpf-grants-secretary at perl-foundation.org.
Test::Simple + Test::Stream stabilization and merge into perl-blead
Stabilize and release the Test-Stream + Test-Simple overhaul.
Benefits to the Perl Community
Test-Simple (an important CPAN distribution which includes Test::Builder and Test::More) is the heart of nearly every test tool for perl. For several years now, the distribution has had very little in the way of progress or development.
For almost two years now, I have been working on revitalizing the project. My work focuses on introducing much needed new features and fixing long-standing issues. I've done all this while maintaining extremely high standards of backwards compatibility.
Here is a list of just some of the benefits:
Improved support for UTF-8 and other encodings
With the current Test::Builder based tools it can be painful to use non-ASCII encodings. The documentation provides an example of iterating over several handles used internally by Test::Builder and changing the encoding one at a time.
With Test::Stream, the encoding can be changed easily, and on the fly.
Better tester-testing capabilities
There are two tools available for testing tools written on Test::Builder. Both of them have major drawbacks and produce very fragile tests.
Test::Stream was written with improved tester-testing - writing tests for the library you've written to write tests - as a primary goal. Writing tests for tools built on Test::Stream is very easy and significantly less fragile.
API enhancements that learn from the 10+ years of Test::Builder history
Test::Builder has been at the core of testing in Perl for over 10 years. In this time, many valuable lessons have been learned, usually in the form of design decisions that were wrong but could not be easily reversed. Some stand-out examples are the singleton, monkey patching, and the subtest implementation.
Test-Stream was developed while taking a hard look at the pain points in Test::Builder. Difficult feature requests and modules forced to use bad-practices to achieve their goals were examined in depth. APIs have been added to address the specific needs that led people to abuse the implementation of the old Test::Builder.
Hooks that provide much-requested capabilities
There are many places in Test::Builder where people have wanted to be able to plug in new functionality. For the most part people had to resort to monkeypatching Test::Builder methods. Monkey patching is not sustainable, and breaks the idea of test-tool interoperability.
Test::Stream is full of hooks, and is written in a way that new hooks can be easily added if needed.
A very complete test suite
Test::Stream has a very rich and complete test suite. Currently it sits at 95.2% coverage, though this is actually not entirely accurate as Devel::Cover is unable to cover thread tests.
Note: High coverage does not mean no bugs, and coverage for its own sake is rarely an effective goal.
Extremely good backwards compatibility
Turning Test::Builder into a compatibility wrapper means that a vast majority of existing tools will continue to work unchanged, and will still play nicely with new tools written on Test::Stream. There are a few exceptions to this, but they are rare on CPAN. Generally only modules which need to completely replace the Test::Builder singleton will be broken. Modules which monkeypatch Test::Builder will typically continue to work so long as they follow good practices.
At this point the project is feature-complete and needs to move from rapid-churn into stabilization. The code seems ready, but I need to battle-test it.
Test-Stream in bleadperl
Test-Stream is the new heart of the testing ecosystem. New tools should be built directly on Test-Stream for access to the latest capabilities. Perl's ""blead"" branch is where the next version of perl is developed. Putting Test-Stream in blead is an effective way to get run on many configurations. =item A Test-Simple powered by Test-Stream in perl-blead
The next Test-Stream will include a Test::Builder that has been modified to defer to Test::Stream under the hood. This will turn Test::Builder into a compatibility wrapper.
Merging this into blead will result in the new code receiving much more testing against CPAN. Blead is smoked against CPAN very frequently, so it is a good place to put these modules to the test.
Completion of the punch-list set by the perl-qa group in Berlin 2015
At the perl-qa hackathon in Berlin (2015) we had a discussion that resulted in a ""punch list"" of requirements for an overhaul of Test-Simple.
The list can be found here: https://gist.github.com/dagolden/2134567dc1125d20d375
A release of Test-Stream without the 'experimental' warnings
A non-dev release of Test-Simple that uses Test-Stream under the hood
This is essentially crossing the finish line, it will effectively put the new code into production.
This grant request is to cover the work needed to finalize what I have and to then release it. There are other aspects of the project that may be part of other grant requests, but those are out of scope for this proposal.
The project started when a seemingly innocent change to Test::Builder was found to break a large chunk of CPAN. The problem had to do with how difficult it was to test testing tools, and how fragile those tests can be. Initially the goal was simply to create a better tool for testing the test tools.
It was quickly (re)discovered that Test::Builder had severe limitations and inadequacies. Fixing these issues was hindered by several design decisions in Test::Builder itself. This kicked off several months of refactoring. The project transitioned through several phases, ultimately landing on a new 'core' in Test::Stream, and modifications of Test::Builder to be a compatibility wrapper around it.
At this point there is almost 2 years of progress, discussion, and contributions to the project. Most of the coding was done by me, but there has been involvement by several others in the form of bug reports, testing, and discussion of ideas. The project has support from the current pumpking (rjbs) and several members of the perl-qa community.
Patches for perl-blead
Previous iterations on the project already did this, and because of that I have a fairly good idea of what is required here (but the previous iterations' patches are not likely to be reusable).
Note: These are not in chronological order, they are organized by category, many of them will be done in parallel.
- Patch to add the Test-Stream CPAN dist
Patch to update to the Test-Simple dev release
These first 2 are fairly simple, and won't take much time.
Patch to update tests that fail due to loaded XS modules
This has proven difficult in the past, it is not always obvious that something is failing because it expects an XS module to either already be loaded, or not loaded.
Patch to update tests that fail due to incompatibilities
Backwards compatibility is a priority, but there are a few cases of things that cannot be supported. I do not recall any of these being in blead, but I will assume they are present until proven otherwise.
Combined patch for inclusion into perl-blead
These patches will all be merged into a single path for perl-blead. Merging the patches makes it easier to do a bisect as it avoids git revisions where tests fail.
This is a set of release criteria laid out during the 2015 perl-QA hackathon in Berlin.
One clear 'master' branch (per dist)
Test-Stream has this, but currently Test-Simple has 2 masters, one for legacy and one for the new. At completion of the project the legacy/master branch will go away, and a single 'master' branch will be added pointing to what is currently stream/master.
Document describing all known issues
We had a document, but it is no longer even remotely accurate, a new doc needs to be written.
Ask people to try it for a period
This will happen as soon as the combined blead patch is ready. A blog post will be written, and shared on several sites.
CPAN Delta smoke
This will be ongoing.
(Maybe) $Test::Builder::Level review
This was an original punch-list item. The requester has since decided he no longer feels the need to do this review, or at least that the review should not be a blocker.
blead-perl verbose delta
This will be a comparison of verbose test output between blead with my patch and blead without my patch. All differences in output need to be explained. If there are unexplained or undesirable differences they must be fixed.
The perl-qa group decided that the new Test-Simple implementation cannot be more than 15% slower than the current stable version. If the 15% limit is exceeded I will need to tune performance until it is below the limit.
Last time I checked there was only an 8% slowdown, well within the limit. There have been many changes since then so I cannot be sure I am still under this limit.
Release Test-Stream without the 'experimental' note in every file
Currently every module in Test-Stream has a giant warning about it being experimental. Removal of this warning will signify production readiness.
Release a non-dev Test-More dist
This will make the new code the default for anyone who upgrades Test-Simple.
Patch perl-blead with the final versions of each dist
Once we have stable releases, a final perl-blead patch will be necessary to reflect the changes. This will be based off the previous patches and should require very little effort.
I can start immediately. I will be working on the project mainly during evening hours and occasional weekend hours.
I expect the project to take several months, easily consuming 80+ hours of time.
- Test-Stream and the updated Test-Simple in a stable perl release
Test-Stream and Test-Simple released as stable on cpan
This is very easy to verify.
Lack of angry hordes storming my house for breaking their tests
My name is Chad Granum. I am known as Exodist on CPAN. I have spent several years writing Perl for work and for fun. For most of my time in Perl I've specialized in unit testing tools. Fennec is a good example of my previous work.
In January of 2014 Michael Schwern transferred the Test-Simple dist and all of its components to me. He felt I was the best person to carry the project forward. Since then I have reached out to the rest of the community to be sure I was carrying the project forward in the best interests of all.
There have been previous attempts to revive the project with rewrites, some of which received grants. Ultimately these attempts fizzled out, and the rewrites were scrapped. This project is a new approach that should not suffer the same demise. For starters this grant request comes well into the project, after almost 2 years of work. Secondly this project did not start as a full rewrite, but rather as a refactor (though it did turn into a partial rewrite).