Test::Builder 2 Final Grant Report

No Comments
Michael Schwern didn't finish his grant in the usual way, completing all the proposed milestones, but he was forced into it, as the Grants Committee decided a hard deadline for the grant (that ha started in 2008!). We asked Schwern a final report, stating what was done, and what is missing. This report will be evaluated by the Grants Committee members, deciding whether Schwern should be paid for the full grant.

The result (payment, or not) is completely dependent on the Grants Committee members will, but nevertheless, we would love to ear the community commenting on what was accomplished, and what is missing, and how worth is Schwern work for having a payment.

Follows the report.


Executive Summary

The bad news first.
  • Test::Builder2 is not yet stable.
  • A handful of important Test:: modules violate the encapsulation of Test::Builder and will need to be fixed.
Now the good news.
  • Test::Builder2 does everything the current Test::Builder can.
  • It is compatible with all documented Test::Builder features.
  • Almost all of the deliverables have been met.
  • The test suite has more than doubled.
  • It now has an extremely flexible system to extend and modify test behavior.
  • The project is in far better shape than it was three years ago.
  • A lot of Test:: modules which are written using messy, hacky encapsulation violations of Test::Builder can now be done elegantly.

Clarification about "Test::Builder2" vs "TB2"

In this document, "Test::Builder2" refers to the new class intended to act like Test::Builder. It has not been completed.

"TB2" will refer to the new set of classes and roles devised to support Test::Builder2 which are now also supporting Test::Builder.

NaMoWriMo


This is my final report to the grant committee on the status of Test::Builder2. It's been a long time coming, and I thank the committee for keeping the grant open all this time. It's helped keep the work going even in the doldrums of major design problems.

When I was informed the grant was going to be shut down, I decided to dedicate November to giving it one last try. A National Module Writing Month. I didn't succeed, but I got a long way.

There have been 282 commits in November alone adding 21588 lines and removing 17597 and we've picked up six new commiters.

The design and development process is now more transparent. Instead of being in my head, it's all written out in Github issues and milestones. https://github.com/schwern/test-more/issues?labels=Test-Builder2 Work is done in branches and reviewed in public.

Test::Builder 1.5

I realized in early November that I could not meet all the features promised in the grant. Instead, I decided to shoot for a stable release. Test::Builder was rewritten using the new TB2 internals creating version 1.5.0.

This Test::Builder 1.5 gives most of the flexibility and features intended for Test::Builder2 without breaking compatibility. It allows us to use and stabilize the new TB2 internals without having to wait for all the new features to be complete.

In that respect it has been a great success. Test::Builder2's new interface and features was a time and energy suck. Focusing on getting Test::Builder's existing interface operational with TB2 rapidly solved many design problems and missing features in TB2.

In the grant I stated that "there's a good chance much of the changes having to do with turning Test::Builder2 into an aggregate will be able to be backported to Test::Builder." It turns out almost ALL of the changes were able to be back ported vastly improving the design of Test::Builder.

The end result is Test::Builder 1.5 is now feature complete. It can do everything the current stable version of Test::Builder can and 100% passes tests. This is a major step towards stability and maturing the TB2 code.

Extensibility

Test::Builder is now flexible far beyond what was intended when the grant was written, and perhaps a bit beyond what is prudent.

Early on I realized that the history of this project has been about giving up control. First I wrote Test::More, and it was good. I controlled what functions went into Test::More. But people wanted to write their own test functions. So chromatic and I wrote Test::Builder, and now people could write and maintain their own test modules. I no longer had say in what people could test, and test modules proliferated. My bad ideas weren't blocking everyone else.

My ideas about Test::Builder were limiting what people could do. Test::Builder2 would also have my biases and limitations. It was time to get out of the way. Instead of just writing Test::Builder2, I pushed the bit I had exclusive control over down another level again.

This eventually resulted in the current system of test events and event handlers. Builders no longer do work, they issue events. These are seen by event handlers and acted upon. An event handler might record events and provide information about what state the test is in (TB2::History), or it might format the event and output it (TB2::Formatter). It might enforce there being only one plan per test (TB2::OnlyOnePlan) or it could capture and test for warnings (Test::NoWarnings).

In this way, a builder just becomes an interface for issuing test events. Anyone can write their own builder. Anyone can write their own event handler. Anyone can change and control the behavior of tests.

Test::Builder2 would inevitably turn out to have limitations in the design, but that's ok, someone can write another.

This radical extensibility also means that Test module authors can do things I never considered. It's already paid off in implementing some of the tricker Test::Builder features like subtests and threads. It also makes a lot of very hacky Test modules far easier. Test::Aggregate, Test::SharedFork and Test::Class will be far cleaner.

Test::Builder held together pretty well for 10 years while the Perl community went from knowing almost nothing about testing to an explosion of testing modules. I hope Test::Builder 1.5 to make it another 10 years with us doing who knows what with tests.

Deliverables

In the grant I promised a number of deliverables. While I tried to write the deliverables in such a way as to be valid no matter how the internals worked out, the design has drifted considerably. So I'll do my best to line up the deliverables with reality.

Here is an accounting:
  • Split up the Test::Builder object into an aggregate of objects.
100% complete.

Test::Builder's responsibilities have been split into...
  • TB2::Formatter (producing formatted output)
  • TB2::Result (representing test results)
  • TB2::History (storing test state and counter)
  • TB2::Event (representing non-result events, like setting the plan)
  • TB2::EventHandler (flexible handling of results and events)
  • TB2::TestState (allows multiple "tests" to happen cleanly at the same time)
  • TB2::Streamer (displays formatted output)
  • Split up global shared Test resources into individual objects
100% complete.

Test::Builder itself now contains (almost) no state of the test itself. It is a convenient interface for setting up the test, issuing test events, and producing results. All shared resources and data about the test reside in the objects listed above.

By sharing TB2::TestState and passing events to it, multiple builders, written by different authors, can participate in the same test as equals. This means a 3rd party can write their own version of Test::Builder using different ideas about the interface and modules written based on this new builder will still work in harmony with Test::Builder based modules.

  • Split up localizable behaviors into objects
100% complete.
As above.
  • Allow individual test modules to locally override Test::Builder2 behaviors
100% complete.

It turned out after some analysis there wasn't much need for "localized" changes to test behavior. But it is possible by swapping in and out EventHandlers or even the entire TestState. An example of this in action is Test::Builder::Tester which has to swap between capturing tests and issuing tests. It is currently being rewritten to accomplish this by simply swapping out the TestState.
  • Allow test modules to globally override Test::Builder2 behaviors
100% complete.

This is accomplished by Test::Builder issuing events for everything which happens during the test. Event handlers do the work. These handlers can be swapped out and custom handlers written.
  • Allow test modules to change how the plan works
100% complete.

There is a "plan" event. The formatter (and event handler) decides what to do with it. Guarding against multiple plans is handled by another event handler, TB2::OnlyOnePlan. It can be removed or replaced if desired.

A potential user of this would be Test::Aggregate. It would add an "early" watcher which would intercept plan events and prevent them from being seen by other handlers. This would allow multiple .t files to be chained together.
  • Allow hooks for global beginning and end of test functions.
Partially complete.

There are hooks for when a test happens, but not for the beginning and the end. This eliminates a lot of modules having to put wrappers around Test::Builder->ok.

This was one of the major reasons for the whole rewrite. Test::Builder cannot identify when a test function ends and so there can be no end-of-test actions. Test::Builder2 was written to address this shortcoming. Unfortunately, the design is incomplete and was put aside to get Test::Builder 1.5 working.

Test::Builder2 is still possible. It's also possible somebody else will write an even better new builder.
  • Ensure multiple hooks "stack"
100% complete
Event handlers stack and have an ordering. They can even modify events.
  • die on fail
Partially complete.

This is one of the major reasons for the rewrite.

There is a proof of concept: https://github.com/schwern/test-more/blob/Test-Builder1.5/examples/TB2/lib/TB2/DieOnFail.pm

It dies in the middle of the test routine rather than at the end. Test::Builder2 addresses this problem, but it is incomplete.
  • debug on fail
Partially complete.

Same as die on fail. Proof of concept: https://github.com/schwern/test-more/blob/Test-Builder1.5/examples/TB2/lib/TB2/DebugOnFail.pm
  • Hooks for global beginning and end of test actions
100% complete
Event handlers can be written to cover these possibilities.
  • Example: A safer Test::NoWarnings
100% complete
This is one of the major reasons for the rewrite.

Possible by writing a handler. When it sees a test_start, it starts capturing warnings. When it sees a plan, it adds one to the number of tests. When testing ends, it issues a failing test if there were warnings.

Proof of concept: https://github.com/schwern/test-more/blob/Test-Builder1.5/examples/TB2/lib/TB2/NoWarnings.pm
  • Example: Don't cleanup temp files on failure so they can be debugged
100% complete

To accomplish this, an event handler watches for a test_end event. At that point, it asks the History object if the test passed or failed. If it passed, it removes the temp files. If it failed, it leaves them in place and informs the user of their location.
  • Example: Allow for test output other than TAP
100% complete

All output is now controlled by TB2::Formatter objects. There are several flavors of TAP formatters (version 12, version 13, and a legacy formatter to match existing quirks).

There's proof of concepts for POSIX and HTML formats to test out formats which are very different from TAP. There's a formatter which outputs nothing and one which can output to multiple formats and locations at once. https://github.com/schwern/test-more/tree/Test-Builder1.5/lib/TB2/Formatter https://github.com/schwern/test-more/tree/Test-Builder1.5/examples/TB2/lib/TB2/Formatter

JSON and jUnit formatters are in the works. https://github.com/schwern/test-more/issues/159 https://github.com/schwern/test-more/issues/158

  • Allow another Test::Builder-like module to work in the same process as Test::Builder (for example, sharing the counter). 
100% complete

All information about the test now resides in TB2::TestState along with all the handlers which perform actions. Two builders can issue events to the same TB2::TestState object and coordinate their work.

This allows anyone to write their own builder and be on equal footing as Test::Builder. It puts the power previously held by me into the hands of any Perl developer. It allows them to rectify any bad decisions in the Test::Builder design.
  • Rewrite Test::Builder in terms of Test::Builder2.
100% complete
Except it's rewritten in terms of TB2. Same effect, but the design concept changed.

Project Schedule

Heh, that was a long five months.

Of the "three major stages", two are complete. There have been nine alpha releases (2.00_01 .. 2.00_07 and 1.5.0_001 .. 1.5.0_002) and Test::Builder has been ported to Test::Builder2.

The idea that "an alpha release can be coded quickly to begin getting feedback and have the API worked with" did not work out. It was not until this month that the design stabilized enough for people to start working with it seriously. I wound up working with little feedback for most of the project. In retrospect, taking the Test::Builder1.5 approach from the start would have gotten a workable module and faster feedback, though I don't know if the resulting redesign would have been as ambitious.

The process of stabilizing the TB2 API and feature set remains. Fortunately, a stable 1.5 release can be made without stabilizing the TB2 API. The new TB2 classes will simply be marked as API unstable.

Road To Stable

Test::Builder 1.5 is not yet stable for several reasons.

  • There's about a dozen small cleanup tasks which need to be done.
The issues remaining before stable are tracked here. https://github.com/schwern/test-more/issues?milestone=6
  • There are several critical CPAN modules which have been broken.
The CPAN modules are generally broken for two reasons:

  1. There was a subtle change in an undocumented Test::Builder feature. Example: POE https://github.com/schwern/test-more/issues/131
  2. The module grossly violated Test::Builder's encapsulation, but there wasn't a better way to do it. Example: Test::Aggregate https://github.com/schwern/test-more/issues/98
In both cases I'm collecting lists of broken modules and working with the authors to fix them.

The community needs time to knock it around. This is a 10 year old module whose guts have just been torn out and replaced. It's used by 80% of CPAN on every platform, in every situation and across 8 years of Perl releases. There's the potential for a lot of breakage, and there's a lot of very subtle things which have to be gotten just right.

Conclusion

The work done has spilled over its original conception when the grant was written. The design bears little resemblance to what was in my mind in 2008 and has gone through several major rewrites and improvements, with feedback from many people. The project is about seven times over schedule, but that time was useful for the design to mature.

Even though there is no stable release as promised in the grant, and some deliverables are not 100%, I feel I have greatly exceeded the bounds of the grant in other areas. Test::Builder is now far more extensible than was ever intended.

I'm asking the committee to grant me the remaining funds. Work on a stable Test::Builder will continue.

Thanks for your patience.

Leave a comment

About TPF

The Perl Foundation - supporting the Perl community since 2000. Find out more at www.perlfoundation.org.

About this Entry

This page contains a single entry by Alberto Simões published on December 3, 2011 9:06 PM.

Hague Grant Application: PL/Perl6 Infrastructure and Improvements was the previous entry in this blog.

Improving Perl 5 Grant Extended is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

OpenID accepted here Learn more about OpenID
Powered by Movable Type 4.38