Grant Proposal: Test::Simple + Test::Stream stabilization and merge into perl-blead

Mon, 16-Nov-2015 by Makoto Nozaki  edit post

_We have received the following grant application "**Test::Simple + Test::Stream stabilization and merge into perl-blead**". Please leave feedback in the comments field by November 27th, 2015. If your comment does not appear in 24 hours, contact me at tpf-grants-secretary at perl-foundation.org._ * * * # Test::Simple + Test::Stream stabilization and merge into perl-blead - Name: Chad Granum - Amount Requested: USD 2,500 ## Synopsis Stabilize and release the Test-Stream + Test-Simple overhaul. ## Benefits to the Perl Community Test-Simple (an important CPAN distribution which includes Test::Builder and Test::More) is the heart of nearly every test tool for perl. For several years now, the distribution has had very little in the way of progress or development. For almost two years now, I have been working on revitalizing the project. My work focuses on introducing much needed new features and fixing long-standing issues. I've done all this while maintaining extremely high standards of backwards compatibility. Here is a list of just some of the benefits: - Improved support for UTF-8 and other encodings With the current Test::Builder based tools it can be painful to use non-ASCII encodings. The documentation provides an example of iterating over several handles used internally by Test::Builder and changing the encoding one at a time. With Test::Stream, the encoding can be changed easily, and on the fly. - Better tester-testing capabilities There are two tools available for testing tools written on Test::Builder. Both of them have major drawbacks and produce very fragile tests. Test::Stream was written with improved tester-testing - writing tests for the library you've written to write tests - as a primary goal. Writing tests for tools built on Test::Stream is very easy and significantly less fragile. - API enhancements that learn from the 10+ years of Test::Builder history Test::Builder has been at the core of testing in Perl for over 10 years. In this time, many valuable lessons have been learned, usually in the form of design decisions that were wrong but could not be easily reversed. Some stand-out examples are the singleton, monkey patching, and the subtest implementation. Test-Stream was developed while taking a hard look at the pain points in Test::Builder. Difficult feature requests and modules forced to use bad-practices to achieve their goals were examined in depth. APIs have been added to address the specific needs that led people to abuse the implementation of the old Test::Builder. - Hooks that provide much-requested capabilities There are many places in Test::Builder where people have wanted to be able to plug in new functionality. For the most part people had to resort to monkeypatching Test::Builder methods. Monkey patching is not sustainable, and breaks the idea of test-tool interoperability. Test::Stream is full of hooks, and is written in a way that new hooks can be easily added if needed. - A very complete test suite Test::Stream has a very rich and complete test suite. Currently it sits at 95.2% coverage, though this is actually not entirely accurate as Devel::Cover is unable to cover thread tests. __Note:__ High coverage does not mean no bugs, and coverage for its own sake is rarely an effective goal. - Extremely good backwards compatibility Turning Test::Builder into a compatibility wrapper means that a vast majority of existing tools will continue to work unchanged, and will still play nicely with new tools written on Test::Stream. There are a few exceptions to this, but they are rare on CPAN. Generally only modules which need to completely replace the Test::Builder singleton will be broken. Modules which monkeypatch Test::Builder will typically continue to work so long as they follow good practices. At this point the project is feature-complete and needs to move from rapid-churn into stabilization. The code seems ready, but I need to battle-test it. ## Deliverables - Test-Stream in bleadperl Test-Stream is the new heart of the testing ecosystem. New tools should be built directly on Test-Stream for access to the latest capabilities. Perl's ""blead"" branch is where the next version of perl is developed. Putting Test-Stream in blead is an effective way to get run on many configurations. =item A Test-Simple powered by Test-Stream in perl-blead The next Test-Stream will include a Test::Builder that has been modified to defer to Test::Stream under the hood. This will turn Test::Builder into a compatibility wrapper. Merging this into blead will result in the new code receiving much more testing against CPAN. Blead is smoked against CPAN very frequently, so it is a good place to put these modules to the test. - Completion of the punch-list set by the perl-qa group in Berlin 2015 At the perl-qa hackathon in Berlin (2015) we had a discussion that resulted in a ""punch list"" of requirements for an overhaul of Test-Simple. The list can be found here: https://gist.github.com/dagolden/2134567dc1125d20d375 - A release of Test-Stream without the 'experimental' warnings - A non-dev release of Test-Simple that uses Test-Stream under the hood This is essentially crossing the finish line, it will effectively put the new code into production. ## Project Details This grant request is to cover the work needed to finalize what I have and to then release it. There are other aspects of the project that may be part of other grant requests, but those are out of scope for this proposal. The project started when a seemingly innocent change to Test::Builder was found to break a large chunk of CPAN. The problem had to do with how difficult it was to test testing tools, and how fragile those tests can be. Initially the goal was simply to create a better tool for testing the test tools. It was quickly (re)discovered that Test::Builder had severe limitations and inadequacies. Fixing these issues was hindered by several design decisions in Test::Builder itself. This kicked off several months of refactoring. The project transitioned through several phases, ultimately landing on a new 'core' in Test::Stream, and modifications of Test::Builder to be a compatibility wrapper around it. At this point there is almost 2 years of progress, discussion, and contributions to the project. Most of the coding was done by me, but there has been involvement by several others in the form of bug reports, testing, and discussion of ideas. The project has support from the current pumpking (rjbs) and several members of the perl-qa community. ## Inch-stones ### Patches for perl-blead Previous iterations on the project already did this, and because of that I have a fairly good idea of what is required here (but the previous iterations' patches are not likely to be reusable). __Note:__ These are not in chronological order, they are organized by category, many of them will be done in parallel. - Patch to add the Test-Stream CPAN dist - Patch to update to the Test-Simple dev release These first 2 are fairly simple, and won't take much time. - Patch to update tests that fail due to loaded XS modules This has proven difficult in the past, it is not always obvious that something is failing because it expects an XS module to either already be loaded, or not loaded. - Patch to update tests that fail due to incompatibilities Backwards compatibility is a priority, but there are a few cases of things that cannot be supported. I do not recall any of these being in blead, but I will assume they are present until proven otherwise. - Combined patch for inclusion into perl-blead These patches will all be merged into a single path for perl-blead. Merging the patches makes it easier to do a bisect as it avoids git revisions where tests fail. ### Punch List This is a set of release criteria laid out during the 2015 perl-QA hackathon in Berlin. - One clear 'master' branch (per dist) Test-Stream has this, but currently Test-Simple has 2 masters, one for legacy and one for the new. At completion of the project the legacy/master branch will go away, and a single 'master' branch will be added pointing to what is currently stream/master. - Document describing all known issues We had a document, but it is no longer even remotely accurate, a new doc needs to be written. - Ask people to try it for a period This will happen as soon as the combined blead patch is ready. A blog post will be written, and shared on several sites. - CPAN Delta smoke This will be ongoing. - (Maybe) $Test::Builder::Level review This was an original punch-list item. The requester has since decided he no longer feels the need to do this review, or at least that the review should not be a blocker. - blead-perl verbose delta This will be a comparison of verbose test output between blead with my patch and blead without my patch. All differences in output need to be explained. If there are unexplained or undesirable differences they must be fixed. - Performance Tuning The perl-qa group decided that the new Test-Simple implementation cannot be more than 15% slower than the current stable version. If the 15% limit is exceeded I will need to tune performance until it is below the limit. Last time I checked there was only an 8% slowdown, well within the limit. There have been many changes since then so I cannot be sure I am still under this limit. ### Release - Release Test-Stream without the 'experimental' note in every file Currently every module in Test-Stream has a giant warning about it being experimental. Removal of this warning will signify production readiness. - Release a non-dev Test-More dist This will make the new code the default for anyone who upgrades Test-Simple. - Patch perl-blead with the final versions of each dist Once we have stable releases, a final perl-blead patch will be necessary to reflect the changes. This will be based off the previous patches and should require very little effort. ## Project Schedule I can start immediately. I will be working on the project mainly during evening hours and occasional weekend hours. I expect the project to take several months, easily consuming 80+ hours of time. ## Completeness Criteria - Test-Stream and the updated Test-Simple in a stable perl release - Test-Stream and Test-Simple released as stable on cpan This is very easy to verify. - Lack of angry hordes storming my house for breaking their tests :-) ## Bio My name is Chad Granum. I am known as Exodist on CPAN. I have spent several years writing Perl for work and for fun. For most of my time in Perl I've specialized in unit testing tools. [Fennec](https://metacpan.org/pod/Fennec) is a good example of my previous work. In January of 2014 Michael Schwern transferred the Test-Simple dist and all of its components to me. He felt I was the best person to carry the project forward. Since then I have reached out to the rest of the community to be sure I was carrying the project forward in the best interests of all. ## Other notes There have been previous attempts to revive the project with rewrites, some of which received grants. Ultimately these attempts fizzled out, and the rewrites were scrapped. This project is a new approach that should not suffer the same demise. For starters this grant request comes well into the project, after almost 2 years of work. Secondly this project did not start as a full rewrite, but rather as a refactor (though it did turn into a partial rewrite).

---

Comments (9)

---

[ 1628 ] | Tue, 17-Nov-2015 by drolsky

I think Test::Stream is an important project for the Perl community, and I think this is a great grant to fund.

---

[ 1629 ] | Tue, 17-Nov-2015 by kentfredric

As much as I appreciate the intent to "Take Test::More forward", I'm still very fearful about the code not having anywhere near as much oversight as it needs.

That is, very few people are actually reviewing the code.

I myself attempted to review it, and I was constantly at a disadvantage due to the pace.

As such, I feel uncomfortable with the plan as-is, because it doesn't make any guarantees about quality.

And we need a more comprehensive strategy that focuses on the review aspects of the code before we throw it into core and unleash it on the world.

Yes, we've got testing, and testing for ages, but testing only compliments code review, it does not replace it.

The rate of change in the codebase to meet the quality required to actually transition needs to slow down, and basically stop for anything other than pure bug fixes and code-review concerns for a long time before we underwrite the universe with it.

I'm not opposing any funding, but I do feel the absence of any kind of review plan for something this important is planning to do the least important things and ignoring the most important.

Sadly, we can't just magically create reviewers who are willing to review the code and dissect it.

---

[ 1630 ] | Tue, 17-Nov-2015 by ether_karenetheridge_

I am concerned to see "Test-Stream in bleadperl" as a deliverable for this grant, as there are prerequisites to that that aren't necessarily covered in this grant.

I would prefer to see the project split up into more components, including a review of the proposed API, as well as an architectural review of the implementation. As it is, the Test-Stream distribution has too great a scope to be integrated into core, and should be split into multiple distributions.

---

[ 1631 ] | Tue, 17-Nov-2015 by ronsavage

Kent's comments are significant, but I still think the project should be funded. Integration into Perl could be delayed to delivery of stable code.

I'm already using Test::Stream and find its error reporting style extremely useful.

---

[ 1632 ] | Wed, 18-Nov-2015 by joelberger

Chad's work is a much needed improvement to the existing test infrastructure. This is especially evident in the mechanisms for testing the test modules written with Test::Stream. Yes the project needs stabilization and oversight but those are things that are part of the objectives of the grant. Of all the module-level projects I've seen come past this committee process, I think this is among the most clearly worth funding and supporting in other ways that this grant can facilitate.

---

[ 1633 ] | Wed, 18-Nov-2015 by https___wp_colliertech_org_cj_

there seems to be a misunderstanding on what 'in perl blead' means. 'in perl blead' in this case does not mean in the next version of perl to be released. It means the code is put into the blead branch so it can get heavy smoking, and lots of eyes on it looking for problems. This has already been done once before, last time it was deemed not ready, so it was not put into the next version on perl. That can happen again.

The goal in this proposal is to go through the process of getting this checked again now that it is more mature.

---

[ 1634 ] | Fri, 20-Nov-2015 by andylester

I'm very concerned about backward compatibility. Something with changes as large as "I'm removing cmp_ok() and changing behavior of is()" seems pretty large. I know that that's changed a bit, but my point is still there.

This is great work, but the chances for breakage are huge. I agree with Karen's direction of "don't make blead part of this grant" while still funding the work.

---

[ 1635 ] | Mon, 23-Nov-2015 by https___wp_colliertech_org_cj_

Andy's comment highlights some confusion. I am not removing or changing anything in Test::More. What I have done is written a new testing infrastructure that solves the problems of Test::Builder. On top of this new infrastructure I have built a new toolset that is intended to be a successor to Test::More, which means it has differences, some tools are new, some are gone, and some work different. The key is that these differences are in a new namespace, they do not effect Test::More.

This grant request is for making Test::Builder use the new infrastructure I have written. This work is important for 3 reasons, 1) it allows tools built on both infrastructures to work together. 2) it allows old tools to take advantage of the new benefits. 3) being in blead ensures it gets more testing on more platforms to ensure nothing breaks.

If the perl blead work is removed from the grant then there is effectively no grant request anymore. I already have a dev release on cpan with the Test::Builder guts altered. All I have left are putting it into blead, fixing bugs that finds, and completing the punch list, which contains several items requiring a merge into blead (or at least the patch, if not an acceptance of the patch). For example, I need to be sure there are no significant test output differences between blead with and without my changes, that requires a complete and working blead patch. Same for the punchlist item asking me to ensure performance is acceptible.

There is fear about letting my work into blead. An earlier version of the work was already put into blead. It was removed before a stable release because it was not ready. Merging into blead unblocks a lot of important work, but in no way forces anyone to release it as core next perl release.

---

[ 1636 ] | Mon, 23-Nov-2015 by sawyerx

Test-Simple is one of the core pieces of CPAN and has been lagging behind for a very long time. Chad has been pushing it forward and he does it openly and transparently.

I think this is one of the most important grants we can provide that has a net-worth for Perl. It affects the stability of CPAN, its testing spirit, the ability to move forward and to address problems we're already trying to work around.

I would like to see this grant approved.

---