Test::Builder 2 Final Grant Report
Sat, 03-Dec-2011 by
Alberto Simões
edit post
<![CDATA[
Michael Schwern didn't finish his grant in the usual way, completing all the proposed milestones, but he was forced into it, as the Grants Committee decided a hard deadline for the grant (that ha started in 2008!). We asked Schwern a final report, stating what was done, and what is missing. This report will be evaluated by the Grants Committee members, deciding whether Schwern should be paid for the full grant.<br /><br />The result (payment, or not) is completely dependent on the Grants Committee members will, but nevertheless, we would love to ear the community commenting on what was accomplished, and what is missing, and how worth is Schwern work for having a payment.<br /><br />Follows the report.<br /><br /><br />
<b>Executive Summary
</b><br />
<br />
The bad news first.
<br />
<ul><li> Test::Builder2 is not yet stable.</li><li>A handful of important Test:: modules violate the encapsulation
of Test::Builder and will need to be fixed.</li></ul>
Now the good news.
<br />
<ul><li>Test::Builder2 does everything the current Test::Builder can.</li><li>It is compatible with all documented Test::Builder features.</li><li>Almost all of the deliverables have been met.</li><li>The test suite has more than doubled.</li><li>It now has an extremely flexible system to extend and modify
test behavior.</li><li>The project is in far better shape than it was three years ago.</li><li>A lot of Test:: modules which are written using messy, hacky
encapsulation violations of Test::Builder can now be done
elegantly.</li></ul><br /><b>Clarification about "Test::Builder2" vs "TB2"</b>
<br /><br />In this document, "Test::Builder2" refers to the new class intended to act like Test::Builder. It has not been completed.
<br /><br />"TB2" will refer to the new set of classes and roles devised to support Test::Builder2 which are now also supporting Test::Builder.
<b><br /><br />NaMoWriMo</b>
<br /><br />This is my final report to the grant committee on the status of
Test::Builder2. It's been a long time coming, and I thank the committee for
keeping the grant open all this time. It's helped keep the work going even in
the doldrums of major design problems.
<br /><br />When I was informed the grant was going to be shut down, I decided to dedicate
November to giving it one last try. A National Module Writing Month. I
didn't succeed, but I got a long way.
<br /><br />There have been 282 commits in November alone adding 21588 lines and removing
17597 and we've picked up six new commiters.
<br /><br />The design and development process is now more transparent. Instead of being
in my head, it's all written out in Github issues and milestones.
<a class="moz-txt-link-freetext" href="https://github.com/schwern/test-more/issues?labels=Test-Builder2">https://github.com/schwern/test-more/issues?labels=Test-Builder2</a>
Work is done in branches and reviewed in public.
<br /><br /><b>Test::Builder 1.5</b><br /><br />I realized in early November that I could not meet all the features promised
in the grant. Instead, I decided to shoot for a stable release.
Test::Builder was rewritten using the new TB2 internals creating version 1.5.0.
<br /><br />This Test::Builder 1.5 gives most of the flexibility and features intended for
Test::Builder2 without breaking compatibility. It allows us to use and
stabilize the new TB2 internals without having to wait for all the new
features to be complete.
<br /><br />In that respect it has been a great success. Test::Builder2's new interface
and features was a time and energy suck. Focusing on getting Test::Builder's
existing interface operational with TB2 rapidly solved many design problems
and missing features in TB2.
<br /><br />In the grant I stated that "there's a good chance much of the changes having
to do with turning Test::Builder2 into an aggregate will be able to be
backported to Test::Builder." It turns out almost ALL of the changes were
able to be back ported vastly improving the design of Test::Builder.
<br /><br />The end result is Test::Builder 1.5 is now feature complete. It can do
everything the current stable version of Test::Builder can and 100% passes
tests. This is a major step towards stability and maturing the TB2 code.
<br /><br /><b>Extensibility
</b><br /><br />Test::Builder is now flexible far beyond what was intended when the grant was
written, and perhaps a bit beyond what is prudent.
<br /><br />Early on I realized that the history of this project has been about giving up
control. First I wrote Test::More, and it was good. I controlled what
functions went into Test::More. But people wanted to write their own test
functions. So chromatic and I wrote Test::Builder, and now people could write
and maintain their own test modules. I no longer had say in what people could
test, and test modules proliferated. My bad ideas weren't blocking everyone else.
<br /><br />My ideas about Test::Builder were limiting what people could do.
Test::Builder2 would also have my biases and limitations. It was time to get
out of the way. Instead of just writing Test::Builder2, I pushed the bit I
had exclusive control over down another level again.
<br /><br />This eventually resulted in the current system of test events and event
handlers. Builders no longer do work, they issue events. These are seen by
event handlers and acted upon. An event handler might record events and
provide information about what state the test is in (TB2::History), or it
might format the event and output it (TB2::Formatter). It might enforce there
being only one plan per test (TB2::OnlyOnePlan) or it could capture and test
for warnings (Test::NoWarnings).
<br /><br />In this way, a builder just becomes an interface for issuing test events.
Anyone can write their own builder. Anyone can write their own event handler.
Anyone can change and control the behavior of tests.
<br /><br />Test::Builder2 would inevitably turn out to have limitations in the design,
but that's ok, someone can write another.
<br /><br />This radical extensibility also means that Test module authors can do things I
never considered. It's already paid off in implementing some of the tricker
Test::Builder features like subtests and threads. It also makes a lot of very
hacky Test modules far easier. Test::Aggregate, Test::SharedFork and
Test::Class will be far cleaner.
<br /><br />Test::Builder held together pretty well for 10 years while the Perl community
went from knowing almost nothing about testing to an explosion of testing
modules. I hope Test::Builder 1.5 to make it another 10 years with us doing
who knows what with tests.
<br /><br /><b>Deliverables
</b><br /><br />In the grant I promised a number of deliverables. While I tried to write the
deliverables in such a way as to be valid no matter how the internals worked
out, the design has drifted considerably. So I'll do my best to line up the
deliverables with reality.
<br /><br />Here is an accounting:<br /><ul><li>Split up the Test::Builder object into an aggregate of objects.</li></ul><blockquote>100% complete.
<br /><br />Test::Builder's responsibilities have been split into...
<br /></blockquote><blockquote><ul><li>TB2::Formatter (producing formatted output)</li><li>TB2::Result (representing test results)</li><li>TB2::History (storing test state and counter)</li><li>TB2::Event (representing non-result events, like setting the plan)</li><li>TB2::EventHandler (flexible handling of results and events)</li><li>TB2::TestState (allows multiple "tests" to happen cleanly at the same time)</li><li>TB2::Streamer (displays formatted output)</li></ul></blockquote><ul><li>Split up global shared Test resources into individual objects</li></ul><blockquote>100% complete.
<br /><br />Test::Builder itself now contains (almost) no state of the test itself. It is
a convenient interface for setting up the test, issuing test events, and
producing results. All shared resources and data about the test reside in the
objects listed above.
<br /><br />By sharing TB2::TestState and passing events to it, multiple builders, written
by different authors, can participate in the same test as equals. This means
a 3rd party can write their own version of Test::Builder using different ideas
about the interface and modules written based on this new builder will still
work in harmony with Test::Builder based modules.
<br /><br /></blockquote><ul><li>Split up localizable behaviors into objects</li></ul><blockquote>100% complete.
<br />As above.
<br /></blockquote><ul><li>
Allow individual test modules to locally override Test::Builder2 behaviors</li></ul><blockquote>100% complete.
<br /><br />It turned out after some analysis there wasn't much need for "localized"
changes to test behavior. But it is possible by swapping in and out
EventHandlers or even the entire TestState. An example of this in action is
Test::Builder::Tester which has to swap between capturing tests and issuing
tests. It is currently being rewritten to accomplish this by simply swapping
out the TestState.
<br /></blockquote><ul><li>
Allow test modules to globally override Test::Builder2 behaviors</li></ul><blockquote>100% complete.<br /><br />This is accomplished by Test::Builder issuing events for everything which
happens during the test. Event handlers do the work. These handlers can be
swapped out and custom handlers written.
</blockquote><ul><li>
Allow test modules to change how the plan works <br /></li></ul><blockquote>100% complete.<br /><br />There is a "plan" event. The formatter (and event handler) decides what to do
with it. Guarding against multiple plans is handled by another event handler,
TB2::OnlyOnePlan. It can be removed or replaced if desired.<br /><br />A potential user of this would be Test::Aggregate. It would add an "early"
watcher which would intercept plan events and prevent them from being seen by
other handlers. This would allow multiple .t files to be chained together.
<br /></blockquote><ul><li>
Allow hooks for global beginning and end of test functions.</li></ul><blockquote>Partially complete.
<br /><br />There are hooks for when a test happens, but not for the beginning and the
end. This eliminates a lot of modules having to put wrappers around
Test::Builder->ok.
<br /><br />This was one of the major reasons for the whole rewrite. Test::Builder cannot
identify when a test function ends and so there can be no end-of-test actions.
Test::Builder2 was written to address this shortcoming. Unfortunately, the
design is incomplete and was put aside to get Test::Builder 1.5 working.
<br /><br />Test::Builder2 is still possible. It's also possible somebody else will write
an even better new builder.
<br /></blockquote><ul><li>Ensure multiple hooks "stack"</li></ul><blockquote>100% complete
<br />Event handlers stack and have an ordering. They can even modify events.
<br /></blockquote><ul><li>die on fail</li></ul><blockquote>Partially complete.
<br /><br />This is one of the major reasons for the rewrite.
<br /><br />There is a proof of concept:
<a class="moz-txt-link-freetext" href="https://github.com/schwern/test-more/blob/Test-Builder1.5/examples/TB2/lib/TB2/DieOnFail.pm">https://github.com/schwern/test-more/blob/Test-Builder1.5/examples/TB2/lib/TB2/DieOnFail.pm</a>
<br /><br />It dies in the middle of the test routine rather than at the end.
Test::Builder2 addresses this problem, but it is incomplete.<br /></blockquote><ul><li>debug on fail</li></ul><blockquote>Partially complete.
<br /><br />Same as die on fail. Proof of concept:
<a class="moz-txt-link-freetext" href="https://github.com/schwern/test-more/blob/Test-Builder1.5/examples/TB2/lib/TB2/DebugOnFail.pm">https://github.com/schwern/test-more/blob/Test-Builder1.5/examples/TB2/lib/TB2/DebugOnFail.pm</a>
<br /></blockquote><ul><li>Hooks for global beginning and end of test actions</li></ul><blockquote>100% complete
<br />Event handlers can be written to cover these possibilities.
<br /></blockquote><ul><li>Example: A safer Test::NoWarnings</li></ul><blockquote>100% complete
<br />This is one of the major reasons for the rewrite.
<br /><br />Possible by writing a handler. When it sees a test_start, it starts capturing
warnings. When it sees a plan, it adds one to the number of tests. When
testing ends, it issues a failing test if there were warnings.<br /><br />Proof of concept:
<a class="moz-txt-link-freetext" href="https://github.com/schwern/test-more/blob/Test-Builder1.5/examples/TB2/lib/TB2/NoWarnings.pm">https://github.com/schwern/test-more/blob/Test-Builder1.5/examples/TB2/lib/TB2/NoWarnings.pm</a>
<br /></blockquote><ul><li>
Example: Don't cleanup temp files on failure so they can be debugged</li></ul><blockquote>100% complete
<br /><br />To accomplish this, an event handler watches for a test_end event. At that
point, it asks the History object if the test passed or failed. If it passed,
it removes the temp files. If it failed, it leaves them in place and informs
the user of their location.
<br /></blockquote><ul><li>Example: Allow for test output other than TAP</li></ul><blockquote>100% complete
<br /><br />All output is now controlled by TB2::Formatter objects. There are several
flavors of TAP formatters (version 12, version 13, and a legacy formatter to
match existing quirks).
<br /><br />There's proof of concepts for POSIX and HTML formats to test out formats which
are very different from TAP. There's a formatter which outputs nothing and
one which can output to multiple formats and locations at once.
<a class="moz-txt-link-freetext" href="https://github.com/schwern/test-more/tree/Test-Builder1.5/lib/TB2/Formatter">https://github.com/schwern/test-more/tree/Test-Builder1.5/lib/TB2/Formatter</a>
<a class="moz-txt-link-freetext" href="https://github.com/schwern/test-more/tree/Test-Builder1.5/examples/TB2/lib/TB2/Formatter">https://github.com/schwern/test-more/tree/Test-Builder1.5/examples/TB2/lib/TB2/Formatter</a>
<br /><br />JSON and jUnit formatters are in the works.
<a class="moz-txt-link-freetext" href="https://github.com/schwern/test-more/issues/159">https://github.com/schwern/test-more/issues/159</a>
<a class="moz-txt-link-freetext" href="https://github.com/schwern/test-more/issues/158">https://github.com/schwern/test-more/issues/158</a></blockquote><br /><ul><li>Allow another Test::Builder-like module to work in the same process as
Test::Builder (for example, sharing the counter). </li></ul><blockquote>100% complete
<br /><br />All information about the test now resides in TB2::TestState along with all
the handlers which perform actions. Two builders can issue events to the same
TB2::TestState object and coordinate their work.
<br /><br />This allows <b class="moz-txt-star"><span class="moz-txt-tag"></span>anyone<span class="moz-txt-tag"></span></b> to write their own builder and be on equal footing as
Test::Builder. It puts the power previously held by me into the hands of any
Perl developer. It allows them to rectify any bad decisions in the
Test::Builder design.
<br /></blockquote><ul><li>Rewrite Test::Builder in terms of Test::Builder2.</li></ul><blockquote>100% complete
<br />Except it's rewritten in terms of TB2. Same effect, but the design concept
changed.
<br /><br /></blockquote><b>Project Schedule
</b><br /><br />Heh, that was a long five months.
<br /><br />Of the "three major stages", two are complete. There have been nine alpha
releases (2.00_01 .. 2.00_07 and 1.5.0_001 .. 1.5.0_002) and Test::Builder has
been ported to Test::Builder2.
<br /><br />The idea that "an alpha release can be coded quickly to begin getting feedback
and have the API worked with" did not work out. It was not until this month
that the design stabilized enough for people to start working with it
seriously. I wound up working with little feedback for most of the project.
In retrospect, taking the Test::Builder1.5 approach from the start would have
gotten a workable module and faster feedback, though I don't know if the
resulting redesign would have been as ambitious.
<br /><br />The process of stabilizing the TB2 API and feature set remains. Fortunately,
a stable 1.5 release can be made without stabilizing the TB2 API. The new TB2
classes will simply be marked as API unstable.
<br /><br /><b>Road To Stable</b><br /><br />Test::Builder 1.5 is not yet stable for several reasons.
<br /><br /><ul><li>There's about a dozen small cleanup tasks which need to be done.</li></ul><blockquote>The issues remaining before stable are tracked here.
<a class="moz-txt-link-freetext" href="https://github.com/schwern/test-more/issues?milestone=6">https://github.com/schwern/test-more/issues?milestone=6</a>
<br /></blockquote><ul><li>There are several critical CPAN modules which have been broken.</li></ul><blockquote>The CPAN modules are generally broken for two reasons:
<br /><br /><ol><li>There was a subtle change in an undocumented Test::Builder feature.
Example: POE <a class="moz-txt-link-freetext" href="https://github.com/schwern/test-more/issues/131">https://github.com/schwern/test-more/issues/131</a></li><li>The module grossly violated Test::Builder's encapsulation, but
there wasn't a better way to do it.
Example: Test::Aggregate <a class="moz-txt-link-freetext" href="https://github.com/schwern/test-more/issues/98">https://github.com/schwern/test-more/issues/98</a></li></ol>In both cases I'm collecting lists of broken modules and working with the
authors to fix them.
<br /><br /></blockquote>The community needs time to knock it around.
This is a 10 year old module whose guts have just been torn out and replaced.
It's used by 80% of CPAN on every platform, in every situation and across 8
years of Perl releases. There's the potential for a lot of breakage, and
there's a lot of very subtle things which have to be gotten just right.
<br /><br /><b>Conclusion
</b><br /><br />The work done has spilled over its original conception when the grant was
written. The design bears little resemblance to what was in my mind in 2008
and has gone through several major rewrites and improvements, with feedback
from many people. The project is about seven times over schedule, but that
time was useful for the design to mature.
<br /><br />Even though there is no stable release as promised in the grant, and some deliverables are not 100%, I feel I have greatly exceeded the bounds of the grant in other areas. Test::Builder is now far more extensible than was ever intended.
<br /><br />I'm asking the committee to grant me the remaining funds. Work on a stable Test::Builder will continue.<br /><br />Thanks for your patience.
]]>
Comments (0)
