Grant Proposal: Test::Simple + Test::Stream stabilization and merge into perl-blead
Mon, 16-Nov-2015 by
_We have received the following grant application "**Test::Simple + Test::Stream stabilization and merge into perl-blead**". Please leave feedback in the comments field by November 27th, 2015. If your comment does not appear in 24 hours, contact me at tpf-grants-secretary at perl-foundation.org._
* * *
# Test::Simple + Test::Stream stabilization and merge into perl-blead
- Amount Requested:
Stabilize and release the Test-Stream + Test-Simple overhaul.
## Benefits to the Perl Community
Test-Simple (an important CPAN distribution which includes Test::Builder and
Test::More) is the heart of nearly every test tool for perl. For several years
now, the distribution has had very little in the way of progress or
For almost two years now, I have been working on revitalizing the project. My
work focuses on introducing much needed new features and fixing long-standing
issues. I've done all this while maintaining extremely high standards of
Here is a list of just some of the benefits:
- Improved support for UTF-8 and other encodings
With the current Test::Builder based tools it can be painful to use non-ASCII
encodings. The documentation provides an example of iterating over several
handles used internally by Test::Builder and changing the encoding one at a
With Test::Stream, the encoding can be changed easily, and on the fly.
- Better tester-testing capabilities
There are two tools available for testing tools written on Test::Builder. Both
of them have major drawbacks and produce very fragile tests.
Test::Stream was written with improved tester-testing - writing tests for the
library you've written to write tests - as a primary goal. Writing tests for
tools built on Test::Stream is very easy and significantly less fragile.
- API enhancements that learn from the 10+ years of Test::Builder history
Test::Builder has been at the core of testing in Perl for over 10 years. In
this time, many valuable lessons have been learned, usually in the form of
design decisions that were wrong but could not be easily reversed. Some
stand-out examples are the singleton, monkey patching, and the subtest
Test-Stream was developed while taking a hard look at the pain points in
Test::Builder. Difficult feature requests and modules forced to use
bad-practices to achieve their goals were examined in depth. APIs have been
added to address the specific needs that led people to abuse the implementation
of the old Test::Builder.
- Hooks that provide much-requested capabilities
There are many places in Test::Builder where people have wanted to be able to
plug in new functionality. For the most part people had to resort to
monkeypatching Test::Builder methods. Monkey patching is not sustainable, and
breaks the idea of test-tool interoperability.
Test::Stream is full of hooks, and is written in a way that new hooks can be
easily added if needed.
- A very complete test suite
Test::Stream has a very rich and complete test suite. Currently it sits at
95.2% coverage, though this is actually not entirely accurate as Devel::Cover
is unable to cover thread tests.
__Note:__ High coverage does not mean no bugs, and coverage for its own sake is
rarely an effective goal.
- Extremely good backwards compatibility
Turning Test::Builder into a compatibility wrapper means that a vast majority
of existing tools will continue to work unchanged, and will still play nicely
with new tools written on Test::Stream. There are a few exceptions to this, but
they are rare on CPAN. Generally only modules which need to completely replace
the Test::Builder singleton will be broken. Modules which monkeypatch
Test::Builder will typically continue to work so long as they follow good
At this point the project is feature-complete and needs to move from
rapid-churn into stabilization. The code seems ready, but I need to battle-test
- Test-Stream in bleadperl
Test-Stream is the new heart of the testing ecosystem. New tools should be
built directly on Test-Stream for access to the latest capabilities. Perl's
""blead"" branch is where the next version of perl is developed. Putting
Test-Stream in blead is an effective way to get run on many configurations.
=item A Test-Simple powered by Test-Stream in perl-blead
The next Test-Stream will include a Test::Builder that has been modified to
defer to Test::Stream under the hood. This will turn Test::Builder into a
Merging this into blead will result in the new code receiving much more testing
against CPAN. Blead is smoked against CPAN very frequently, so it is a good
place to put these modules to the test.
- Completion of the punch-list set by the perl-qa group in Berlin 2015
At the perl-qa hackathon in Berlin (2015) we had a discussion that resulted in
a ""punch list"" of requirements for an overhaul of Test-Simple.
The list can be found here:
- A release of Test-Stream without the 'experimental' warnings
- A non-dev release of Test-Simple that uses Test-Stream under the hood
This is essentially crossing the finish line, it will effectively put the new
code into production.
## Project Details
This grant request is to cover the work needed to finalize what I have and to
then release it. There are other aspects of the project that may be part of
other grant requests, but those are out of scope for this proposal.
The project started when a seemingly innocent change to Test::Builder was found
to break a large chunk of CPAN. The problem had to do with how difficult it was
to test testing tools, and how fragile those tests can be. Initially the goal
was simply to create a better tool for testing the test tools.
It was quickly (re)discovered that Test::Builder had severe limitations and
inadequacies. Fixing these issues was hindered by several design decisions in
Test::Builder itself. This kicked off several months of refactoring. The
project transitioned through several phases, ultimately landing on a new 'core'
in Test::Stream, and modifications of Test::Builder to be a compatibility
wrapper around it.
At this point there is almost 2 years of progress, discussion, and
contributions to the project. Most of the coding was done by me, but there has
been involvement by several others in the form of bug reports, testing, and
discussion of ideas. The project has support from the current pumpking (rjbs)
and several members of the perl-qa community.
### Patches for perl-blead
Previous iterations on the project already did this, and because of that I have
a fairly good idea of what is required here (but the previous iterations'
patches are not likely to be reusable).
__Note:__ These are not in chronological order, they are organized by category,
many of them will be done in parallel.
- Patch to add the Test-Stream CPAN dist
- Patch to update to the Test-Simple dev release
These first 2 are fairly simple, and won't take much time.
- Patch to update tests that fail due to loaded XS modules
This has proven difficult in the past, it is not always obvious that something
is failing because it expects an XS module to either already be loaded, or not
- Patch to update tests that fail due to incompatibilities
Backwards compatibility is a priority, but there are a few cases of things that
cannot be supported. I do not recall any of these being in blead, but I will
assume they are present until proven otherwise.
- Combined patch for inclusion into perl-blead
These patches will all be merged into a single path for perl-blead. Merging the
patches makes it easier to do a bisect as it avoids git revisions where tests
### Punch List
This is a set of release criteria laid out during the 2015 perl-QA hackathon
- One clear 'master' branch (per dist)
Test-Stream has this, but currently Test-Simple has 2 masters, one for legacy
and one for the new. At completion of the project the legacy/master branch will
go away, and a single 'master' branch will be added pointing to what is
- Document describing all known issues
We had a document, but it is no longer even remotely accurate, a new doc needs
to be written.
- Ask people to try it for a period
This will happen as soon as the combined blead patch is ready. A blog post will
be written, and shared on several sites.
- CPAN Delta smoke
This will be ongoing.
- (Maybe) $Test::Builder::Level review
This was an original punch-list item. The requester has since decided he no
longer feels the need to do this review, or at least that the review should not
be a blocker.
- blead-perl verbose delta
This will be a comparison of verbose test output between blead with my patch
and blead without my patch. All differences in output need to be explained. If
there are unexplained or undesirable differences they must be fixed.
- Performance Tuning
The perl-qa group decided that the new Test-Simple implementation cannot be
more than 15% slower than the current stable version. If the 15% limit is
exceeded I will need to tune performance until it is below the limit.
Last time I checked there was only an 8% slowdown, well within the limit. There
have been many changes since then so I cannot be sure I am still under this
- Release Test-Stream without the 'experimental' note in every file
Currently every module in Test-Stream has a giant warning about it being
experimental. Removal of this warning will signify production readiness.
- Release a non-dev Test-More dist
This will make the new code the default for anyone who upgrades Test-Simple.
- Patch perl-blead with the final versions of each dist
Once we have stable releases, a final perl-blead patch will be necessary to
reflect the changes. This will be based off the previous patches and should
require very little effort.
## Project Schedule
I can start immediately. I will be working on the project mainly during evening
hours and occasional weekend hours.
I expect the project to take several months, easily consuming 80+ hours of
## Completeness Criteria
- Test-Stream and the updated Test-Simple in a stable perl release
- Test-Stream and Test-Simple released as stable on cpan
This is very easy to verify.
- Lack of angry hordes storming my house for breaking their tests
My name is Chad Granum. I am known as Exodist on CPAN. I have spent several
years writing Perl for work and for fun. For most of my time in Perl I've
specialized in unit testing tools. [Fennec](https://metacpan.org/pod/Fennec) is a good example of my previous
In January of 2014 Michael Schwern transferred the Test-Simple dist and all of
its components to me. He felt I was the best person to carry the project
forward. Since then I have reached out to the rest of the community to be sure
I was carrying the project forward in the best interests of all.
## Other notes
There have been previous attempts to revive the project with rewrites, some of
which received grants. Ultimately these attempts fizzled out, and the rewrites
were scrapped. This project is a new approach that should not suffer the same
demise. For starters this grant request comes well into the project, after
almost 2 years of work. Secondly this project did not start as a full rewrite,
but rather as a refactor (though it did turn into a partial rewrite).
I think Test::Stream is an important project for the Perl community, and I think this is a great grant to fund.
As much as I appreciate the intent to "Take Test::More forward", I'm still very fearful about the code not having anywhere near as much oversight as it needs.
That is, very few people are actually reviewing the code.
I myself attempted to review it, and I was constantly at a disadvantage due to the pace.
As such, I feel uncomfortable with the plan as-is, because it doesn't make any guarantees about quality.
And we need a more comprehensive strategy that focuses on the review aspects of the code before we throw it into core and unleash it on the world.
Yes, we've got testing, and testing for ages, but testing only compliments code review, it does not replace it.
The rate of change in the codebase to meet the quality required to actually transition needs to slow down, and basically stop for anything other than pure bug fixes and code-review concerns for a long time before we underwrite the universe with it.
I'm not opposing any funding, but I do feel the absence of any kind of review plan for something this important is planning to do the least important things and ignoring the most important.
Sadly, we can't just magically create reviewers who are willing to review the code and dissect it.
I am concerned to see "Test-Stream in bleadperl" as a deliverable for this grant, as there are prerequisites to that that aren't necessarily covered in this grant.
I would prefer to see the project split up into more components, including a review of the proposed API, as well as an architectural review of the implementation. As it is, the Test-Stream distribution has too great a scope to be integrated into core, and should be split into multiple distributions.
Kent's comments are significant, but I still think the project should be funded. Integration into Perl could be delayed to delivery of stable code.
I'm already using Test::Stream and find its error reporting style extremely useful.
Chad's work is a much needed improvement to the existing test infrastructure. This is especially evident in the mechanisms for testing the test modules written with Test::Stream. Yes the project needs stabilization and oversight but those are things that are part of the objectives of the grant. Of all the module-level projects I've seen come past this committee process, I think this is among the most clearly worth funding and supporting in other ways that this grant can facilitate.
there seems to be a misunderstanding on what 'in perl blead' means. 'in perl blead' in this case does not mean in the next version of perl to be released. It means the code is put into the blead branch so it can get heavy smoking, and lots of eyes on it looking for problems. This has already been done once before, last time it was deemed not ready, so it was not put into the next version on perl. That can happen again.
The goal in this proposal is to go through the process of getting this checked again now that it is more mature.
I'm very concerned about backward compatibility. Something with changes as large as "I'm removing cmp_ok() and changing behavior of is()" seems pretty large. I know that that's changed a bit, but my point is still there.
This is great work, but the chances for breakage are huge. I agree with Karen's direction of "don't make blead part of this grant" while still funding the work.
Andy's comment highlights some confusion. I am not removing or changing anything in Test::More. What I have done is written a new testing infrastructure that solves the problems of Test::Builder. On top of this new infrastructure I have built a new toolset that is intended to be a successor to Test::More, which means it has differences, some tools are new, some are gone, and some work different. The key is that these differences are in a new namespace, they do not effect Test::More.
This grant request is for making Test::Builder use the new infrastructure I have written. This work is important for 3 reasons, 1) it allows tools built on both infrastructures to work together. 2) it allows old tools to take advantage of the new benefits. 3) being in blead ensures it gets more testing on more platforms to ensure nothing breaks.
If the perl blead work is removed from the grant then there is effectively no grant request anymore. I already have a dev release on cpan with the Test::Builder guts altered. All I have left are putting it into blead, fixing bugs that finds, and completing the punch list, which contains several items requiring a merge into blead (or at least the patch, if not an acceptance of the patch). For example, I need to be sure there are no significant test output differences between blead with and without my changes, that requires a complete and working blead patch. Same for the punchlist item asking me to ensure performance is acceptible.
There is fear about letting my work into blead. An earlier version of the work was already put into blead. It was removed before a stable release because it was not ready. Merging into blead unblocks a lot of important work, but in no way forces anyone to release it as core next perl release.
Test-Simple is one of the core pieces of CPAN and has been lagging behind for a very long time. Chad has been pushing it forward and he does it openly and transparently.
I think this is one of the most important grants we can provide that has a net-worth for Perl. It affects the stability of CPAN, its testing spirit, the ability to move forward and to address problems we're already trying to work around.
I would like to see this grant approved.