No test plan gets printed
gyles19 opened this issue · comments
New QUnit and QUnit-tap user here. I've started adding some javascript to an existing perl CGI::Application web app. I put together a 'Hello, world' style of unit test based on what I read in your README, then added it to my existing test suites, and it failed because it had no plan.
$ prove t/suite.t
t/suite.t .. All 3 subtests passed
Test Summary Report
t/suite.t (Wstat: 0 Tests: 3 Failed: 0)
Parse errors: No plan found in TAP output
Files=1, Tests=3, 1 wallclock secs ( 0.03 usr 0.01 sys + 1.59 cusr 0.15 csys = 1.78 CPU)
Result: FAIL
In perl's TAP, you must provide a plan line, either at the front of the test, or at the end. By default, with Test::More, you specify the number of tests in the file when you initialize things. Test::More offers a 'no_plan' option where it will keep count for you and output the results at the end of the file, instead of at the beginning as normal.
With QUnit-tap, I'm not getting a plan line in the TAP output at all using the defaults (no_plan:false). I do get an output at the end when I use (no_plan:true.) My test file is specifying the number of tests to QUnit test(), and I've tried it both by passing the expected number to the test() function, and also by calling expects() inside the test. Both attempts failed to print a plan.
I would expect that, as QUnit itself knows what the expected number of tests in the file is, QUnit-tap should be printing this value as its first line of output, at the top of the test run. The no_plan idea from Perl's Test::More only moves the plan line to the end, after Test::More counts for you.
Not having any real experience with QUnit and QUnit-tap, I'm not sure what the intent is with this. In Perl, any script containing tests will output a plan either at the top or the bottom. Is QUnit-tap omitting the plan at the top because it's standard to specify multiple test() functions in a single file? (You might look at Test::Class for how that can be accomplished in a TAP manner in native perl.)
So, to summarize, in Test::More, 'no_plan' doesn't mean there is no plan output at all, it just means the test writer isn't going to provide the count, so Test::More will count and output the plan line at the end. QUnit-tap isn't replicating this behavior properly. QUnit-tap is failing to produce a TAP plan when no_plan is false.
First, thank you for your report!
Is QUnit-tap omitting the plan at the top because it's standard to specify multiple test() functions in a single file?
Yes.
QUnit itself knows what the expected number of tests in the file is
Unfortunately, No.
It's a limitation of current QUnit and QUnit-TAP implementation. QUnit only knows how many assertions expected inside test, not module, nor file, nor whole test suite. And there is no simple, clean and backward-compatible way to get the count from QUnit before test's execution (especially, if expect method is used, expectation count appears when function body is executed). I cannot collect whole expectation count cumulatively before any test execution due to the internal execution model of QUnit.
In contrast, Test::More handles expectation count at file level, prints only once at start or at end of its execution. I think it is reasonable for tools like prove. I've created QUnit-TAP to fill the design gap between QUnit and Test::More.
I also think it's the best way to use expectation count that QUnit takes as arguments of test or expect. However, I think changing QUnit internals causes much impact and decreases simplicity of QUnit usage. JavaScript does not handle files like Perl. Even worse, there is a huge difference of execution models between Browser and Node.
Maybe it's time to count by ourselves. I've created a experimental branch to simulate noPlan: true behavior properly.
QUnit.tap.plan method enables us to declare expectation count at file level, then QUnit-TAP collects counts cumulatively across files if required (especially in PhantomJS environment). Thank you for your report to give me a great hint for this implementation.
If you like it, I'll merge and release this feature as a next minor version 1.3.0. Also I'm going to make noPlan: true as default in 1.3.0 release.
Thanks.
Wow! Thank you for responding. I would be delighted to assist you in TAP
delicacies, as I have a number of perl projects using TAP extensively in
test production. (I have to leave for work now, so this message is not as
compact as I had intended, please forgive me its length. I was in the
middle of editing it and looked at the clock. Eep!)
Perl prove, the command-line test harness/test runner, expects plans only
at the source file level. If one runs prove on an entire directory of
tests, prove understands that each test source file is going to provide a
separate plan/count. Test::Class also implements a expect/plan number,
which one specifies in the same way, but one must also provide a
per-test-function expect as well, and Test::Class will then complain if the
sum of all function expects doesn't match the file expect one provided. (I
reference Test::Class because it is a junit-esque form of TAP producer for
perl, so it may actually be a better reference for you implementation than
Test::More is. IE Test::Class has suite support, Test::More does not.
Test::Class is the producer I prefer for all of my perl projects.) The
key point is that Test::Class and Test::More handle the plan line, not the
test author. The author only provides an assertion count. One confusing
point is that inside tests, perl producers call this number a plan value,
but QUnit is calling it the expect/expects/expected value.
Not having written any serious QUnit tests myself, I would have assumed I
needed to use a similar design to perl's Test::Class.. So, I assumed the
current QUnit/QUnit-tap implementations would require a single test() per
file, with its fixture and plan, and multiple asserts. (This assumption
seems to different for current QUnit practices.) I have the
qunit_tap.config stuff in the suite.js file that loads the test.js file(s).
I think the usual practices with perl-based TAP producers, expressed in how
I currently understand QUnit terms, is that:
- a test source file contains a collection of test() functions, and each
test knows how many assertions it makes - a test source file contains a single module, which emits a plan
summing the assertions in all of its tests. The module may be given an
expect() value. - prove loads and runs every test file/module, and verifies that each
module produces a plan and that the OK/NOT OK counts match that plan.
Test:Class supports fixtures (which Test::More does not.) Test::Class
inspects the perl package and finds all of the test functions, then loads
and runs the module once for each function, ensuring each function gets a
fresh copy of the specified fixture. I think I saw QUnit talking about
fixtures but I haven't gotten far enough to learn if they are fresh for
each test function, or not. I think it is 'not'.
Test::Class implements a suite() object, which is loaded with a collection
of test() objects, each of which know their own plans, or not. If there is
no plan provided anywhere, Test::Class will run each test function, get its
result object, and count the results in that object. It sums all of these
results and produces a plan at the very end of execution. Test::Class will
also run a bare module (no suite objects) and do the same thing if the
module is using no_plan, it just counts the results in the result object
the module returns.
Perls test harness, prove, doesn't actually care how many functions,
modules, suites, etc are in the test.t source file, as long as the output
of the TAP stream contains a single plan. In Test::Class suites,
Test::Class asks each of its members what their plan count is, and the
suite emits a single plan for the entire group. If all members declare
their plan counts, it emits a suite plan at the start of execution,
otherwise it emits the plan at the end of execution, when it has the
results available to sum.
It sounds like perhaps QUnit doesn't have the hooks necessary to implement
plan handling in the same fashion as Test::Class or Test::More? I remember
seeing a list of them in the QUnit API online, I shall go back and look at
them again.
If your intent is to provide a fast way to wrap any existing set of QUnit
files in TAP, you need a way to produce that plan line some where, or it
isn't valid TAP and proper TAP consumers will emit a FAIL because not
seeing the plan line forces the assumption that the test failed to execute
to completion.
I will look at this again today, perhaps I can suggest something that leads
you to a clean and easy solution.
On Tue, Dec 4, 2012 at 1:41 AM, Takuto Wada notifications@github.comwrote:
First, thank you for your report!
Is QUnit-tap omitting the plan at the top because it's standard to specify
multiple test() functions in a single file?Yes.
QUnit itself knows what the expected number of tests in the file is
Unfortunately, No.
It's a limitation of current QUnit and QUnit-TAP implementation. QUnit
only knows how many assertions expected inside test, not module, nor
file, nor whole test suite. And there is no simple, clean and
backward-compatible way to get the count from QUnit before test's
execution (especially, if expect method is used, expectation count
appears when function body is executed). I cannot collect whole expectation
count cumulatively before any test execution due to the internal
execution model of QUnit.In contrast, Test::More handles expectation count at execution level,
prints only once at start or at end of its execution. I think it is
reasonable for tools like prove. I've created QUnit-TAP to fill the
design gap between QUnit and Test::More.I think it's the best way to use expectation count that QUnit takes as
arguments of test or expect. However, I think changing QUnit internals
makes much impact and decrease simplicity of QUnit usage. JavaScript does
not handle files like Perl. Even worse, there is a huge difference of
execution models between Browser and Node.Maybe it's time to count by ourselves. I've created a experimental branch
to simulate noPlan: true behavior properly.no_plan_false branchhttps://github.com/twada/qunit-tap/tree/no_plan_false
QUnit.tap.plan method enables us to declare expectation count at file
level, then QUnit-TAP collects counts cumulatively across files if required
(especially in PhantomJS environment). Thank you for your report to give me
a great hint for this implementation.If you like it, I'll merge and release this feature as a next minor
version 1.3.0. Also I'm going to make noPlan: true as default in 1.3.0
release.Thanks.
—
Reply to this email directly or view it on GitHubhttps://github.com//issues/6#issuecomment-10987196.
First of all, thank you for your comment!
As you imagine, I'm not a Perl Monger, so I don't have enough TAP vocabularies. I'm grad to hear you teach me about that. I spent all of my programming life with Unix and really love Unix philosophy, respects Perl's test culture and TAP format.
Now, I have some corrections about QUnit for you.
- a test source file contains a single module, which emits a plan summing the assertions in all of its tests.
A test source file also contains a collection of module() functions, and each module does NOT know how many assertions (test knows it), and even worse, module does NOT know how many tests in it!
The module may be given an expect() value.
The test may be given an expect() value. Not module.
It sounds like perhaps QUnit doesn't have the hooks necessary to implement plan handling in the same fashion as Test::Class or Test::More?
QUnit provides begin, moduleStart, testStart hooks for this purpose, but these hooks does not have plan value, and QUnit does not allow access to plan value. It would be a hard way to retrieve plan value in current design...
I will look at this again today, perhaps I can suggest something that leads you to a clean and easy solution.
Great to hear that!
FYI: Current experimental implementation of QUnit.tap.plan method in no_plan_false branch simply declares plan value for each file. Before the test starts running (timing matters most), QUnit-TAP collects plan values cumulatively across files if required (especially in headless browser environment like PhantomJS, since browser runs all tests together), then print the plan line at the top of the test run. Please take a look at it and give me your comment!
Thanks.
I'm at work so a long reply will have to wait, but I was expressing Perl
TAB assumptions in QUinit terms, not trying to express how I understand
QUnit to work.
I remember Qunit as a test-done hook, right? Qunit-tap could add a
test-done hook that would simply keep a counter of all dones and issue a
plan using that final number?
That way you could emit a plan line just before qunit exits?
On Tue, Dec 4, 2012 at 9:30 AM, Takuto Wada notifications@github.comwrote:
First of all, thank you for your comment!
As you imagine, I'm not a Perl Monger. So I don't have enough TAP
vocabularies. I'm grad to hear you teach me about that. I spent all of my
programming life with Unix and really love Unix philosophy, respects Perl's
test culture and TAP format.Now, I have some corrections about QUnit for you.
a test source file contains a single module, which emits a plan
summing the assertions in all of its tests.A test source file also contains a collection of module() functions, and
each module does NOT know how many assertions (test knows it), and even
worse, module does NOT know how many tests in it!The module may be given an expect() value.
The test may be given an expect() value. Not module.
It sounds like perhaps QUnit doesn't have the hooks necessary to implement
plan handling in the same fashion as Test::Class or Test::More?QUnit provides begin, moduleStart, testStart hooks for this purpose, but
these hooks does not have plan value, and QUnit does not allow access to
plan value. It would be a hard way to retrieve plan value in current
design...I will look at this again today, perhaps I can suggest something that
leads you to a clean and easy solution.Great to hear that!
FYI: Current experimental implementation of QUnit.tap.plan method in no_plan_false
branch https://github.com/twada/qunit-tap/tree/no_plan_false simply
declares plan value for each file. Before the test starts running
(timing matters most), QUnit-TAP collects plan values cumulatively across
files if required (especially in headless browser environment like
PhantomJS, since browser runs all tests together), then print the plan line
at the top of the test run. Please take a look at it and give me your
comment!Thanks.
—
Reply to this email directly or view it on GitHubhttps://github.com//issues/6#issuecomment-11001261.
Ok, I've started doing some browsing through QUnit itself, and branched out
into looking at JUnit-Logger. That extension adds a number of QUnit
callbacks that you may not have considered. Referring to
https://github.com/jquery/qunit/blob/master/addons/junitlogger/junitlogger.js,
look at QUnit.done, QUnit.moduleStart, QUnit.moduleDone, QUnit.testStart,
and QUnit.testEnd.
If you have a test setup with no_plan=true, those callbacks can keep your
count for you. Then from QUnit.done, print the plan based on the counts
you've kept.
You could also document that TAP-compatible tests should always run with
QUnit.requireExpects should be true.
I did some experimenting and find that inside a test, I can do this:
module("MyLibTest");
test("Adding numbers works", 3,
function() {
// expect(3);
print( "test sees expected = " + QUnit.config.current.expected );
ok( newAddition, "function exists");
equal( newAddition(2,2), 4, "2 + 2 = 4" );
equal( newAddition( 100,0 ), 100, "zero is zero" );
}
);
and it prints:
module: MyLibTest
test: Adding numbers works
test sees expected = 3
ok 1 - function exists
ok 2 - 2 + 2 = 4
ok 3 - zero is zero
My ability to play with this right now is rather hampered because
rhino-debugger demands an X11 host and I'm doing this on a console-only
headless debian system. I'd love to single-step into one of the callbacks
and see what I can actually see. I'm too much of a javascript newbie to
really know offhand.
On Tue, Dec 4, 2012 at 9:41 AM, Joi Ellis gyles19@gmail.com wrote:
I'm at work so a long reply will have to wait, but I was expressing Perl
TAB assumptions in QUinit terms, not trying to express how I understand
QUnit to work.I remember Qunit as a test-done hook, right? Qunit-tap could add a
test-done hook that would simply keep a counter of all dones and issue a
plan using that final number?
That way you could emit a plan line just before qunit exits?On Tue, Dec 4, 2012 at 9:30 AM, Takuto Wada notifications@github.comwrote:
First of all, thank you for your comment!
As you imagine, I'm not a Perl Monger. So I don't have enough TAP
vocabularies. I'm grad to hear you teach me about that. I spent all of my
programming life with Unix and really love Unix philosophy, respects Perl's
test culture and TAP format.Now, I have some corrections about QUnit for you.
a test source file contains a single module, which emits a plan
summing the assertions in all of its tests.A test source file also contains a collection of module() functions,
and each module does NOT know how many assertions (test knows it), and
even worse, module does NOT know how many tests in it!The module may be given an expect() value.
The test may be given an expect() value. Not module.
It sounds like perhaps QUnit doesn't have the hooks necessary to
implement plan handling in the same fashion as Test::Class or Test::More?QUnit provides begin, moduleStart, testStart hooks for this purpose, but
these hooks does not have plan value, and QUnit does not allow access to
plan value. It would be a hard way to retrieve plan value in current
design...I will look at this again today, perhaps I can suggest something that
leads you to a clean and easy solution.Great to hear that!
FYI: Current experimental implementation of QUnit.tap.plan method in no_plan_false
branch https://github.com/twada/qunit-tap/tree/no_plan_false simply
declares plan value for each file. Before the test starts running
(timing matters most), QUnit-TAP collects plan values cumulatively across
files if required (especially in headless browser environment like
PhantomJS, since browser runs all tests together), then print the plan line
at the top of the test run. Please take a look at it and give me your
comment!Thanks.
—
Reply to this email directly or view it on GitHubhttps://github.com//issues/6#issuecomment-11001261.
Thank you for your comment!
If you have a test setup with no_plan=true, those callbacks can keep your count for you. Then from QUnit.done, print the plan based on the counts you've kept.
That's what I'm doing in QUnit-TAP ;-)
QUnit.requireExpects & QUnit.config.current.expected
Seems interesting. Maybe I can take test plan (how many assertions expected) at end of each test execution. Thank you for your research!!
However, one problem arises under the case when running tests across many modules and tests. To satisfy "print plan line at the top once and only once" rule, now QUnit-TAP shouldn't print test line immediately but buffer the lines. Then after all test execution, QUnit-TAP prints plan line at the top, then print rest of all buffered test lines.
Hmm... is it reasonable and respects stream-oriented Unix Philosophy, for example when running with prove -j8? (I really love the -j option)
What do you think?
Actually, the TAP rule is there must be one and only plan line, and it must
be either the first or the last line in the test output.
#!/usr/bin/perl
use Test::More tests => 23; # we know the count, so issue a plan line
first, before any test results are printed.
#!/usr/bin/perl
use Test::More 'no_plan'; # don't know the tests, count what actually runs
and print the plan at the end instead.
I see that qunit passes along an object to the tests that lets them look at
some qunit internals via config.current. I am not able to run the
graphical debugger to single step into the thing and look at what that
config.current object exposes. I am expecting to find the
current.config.expected value visible, which should allow qunit-tap to find
out what's expected for the run.
At the very least you could just have QUnit-tap inspect the result object
with a qunit.done hook, and print a plan at the end, regardless of the
plan/no_plan thing. In other words, if Qunit does not expose the expected
count you need for the test run to print the plan line at the start, just
print it at the end instead. You wouldn't need the no_plan parameter
control in Qunit-tap in that case.
On Thu, Dec 6, 2012 at 4:41 AM, Takuto Wada notifications@github.comwrote:
Thank you for your comment!
If you have a test setup with no_plan=true, those callbacks can keep your
count for you. Then from QUnit.done, print the plan based on the counts
you've kept.That's what I'm doing in QUnit-TAP ;-)
QUnit.requireExpects & QUnit.config.current.expected
Seems interesting. Maybe I can take test plan (how many assertions
expected) at end of each test execution. Thank you for your research!!However, one problem arises under the case when running tests across many
modules and tests. To satisfy "print plan line at the top once and only
once" rule, now QUnit-TAP shouldn't print test line immediately but buffer
the lines. Then after all test execution, QUnit-TAP prints plan line at the
top, then print rest of all buffered test lines.Hmm... is it reasonable and respects stream-oriented Unix Philosophy, for
example when running with prove -j8? (I really love the -j option)What do you think?
—
Reply to this email directly or view it on GitHubhttps://github.com//issues/6#issuecomment-11079766.
I'm so sorry for my late response. I've been busy these days :-<
Your suggestion seems reasonable for me, so I've created another experimental branch named require_expects branch.
If QUnit.config.requireExpects is true, QUnit-TAP uses expected assertion count specified by test or expect as plan instead of actual run count. Please take a look!
Then my next task is graceful deprecation of noPlan option in future version.
Thanks.
Ok, I think I've figured out how to checkout that branch (not being a git
user until now...) I'm still trying to figure out how best to write tests
for my own project, as I'm limited to a CLI interface for the
test/build environment. So far I've got rhino and phantomjs running. I
ran the QUnit-tap tests via a browser and saw a number of failures. I may
take a stab and creating some tests that test QUnit-tap's fail messages and
actually pass. (IE something like the Test::Builder::Tester module in
Perl, one of several modules created for testing test harnesses.)
On Tue, Dec 11, 2012 at 12:39 PM, Takuto Wada notifications@github.comwrote:
I'm so sorry for my late response. I've been busy these days :-<
Your suggestion seems reasonable for me, so I've created another
experimental branch named require_expects branchhttps://github.com/twada/qunit-tap/tree/require_expects.If QUnit.config.requireExpects is true, QUnit-TAP uses expected assertion
count specified by QUnit#test or QUnit#expect as plan instead of actual run
count. Please take a look!Then my next task is graceful deprecation of noPlan option in future
version.Thanks.
—
Reply to this email directly or view it on GitHubhttps://github.com//issues/6#issuecomment-11256287.
Oh, I finally found an online reference to the TAP specification. I expect
you have a copy, the definitive resource, testanything.org, has been
offline for weeks now. I found a slightly older specification here at
CPAN:
http://cpansearch.perl.org/src/ARODLAND/TAP-Spec-Parser-0.10/lib/TAP/Spec/Parser.pm
On Tue, Dec 11, 2012 at 12:39 PM, Takuto Wada notifications@github.comwrote:
I'm so sorry for my late response. I've been busy these days :-<
Your suggestion seems reasonable for me, so I've created another
experimental branch named require_expects branchhttps://github.com/twada/qunit-tap/tree/require_expects.If QUnit.config.requireExpects is true, QUnit-TAP uses expected assertion
count specified by QUnit#test or QUnit#expect as plan instead of actual run
count. Please take a look!Then my next task is graceful deprecation of noPlan option in future
version.Thanks.
—
Reply to this email directly or view it on GitHubhttps://github.com//issues/6#issuecomment-11256287.
I'm so sorry for my late response, again.
Today I installed perlbrew and start testing output conformance to the TAP specification with TAP::Spec::Parser on the require_expects branch. Thank you for your suggestion.
Now I'm going to prepare merge the branch and release next version soon.
You don't need to apologize for anything, we all work full time jobs and
these projects are expected to take longer. I'm on vacation myself.
Cheers!
On Sat, Dec 29, 2012 at 11:13 AM, Takuto Wada notifications@github.comwrote:
I'm so sorry for my late response, again.
Today I installed perlbrew and start testing output conformance to the TAP
specification with TAP::Spec::Parser on the require_expects branchhttps://github.com/twada/qunit-tap/tree/require_expects.
Thank you for your suggestion.Now I'm going to prepare merge the branch and release next version soon.
—
Reply to this email directly or view it on GitHubhttps://github.com//issues/6#issuecomment-11755365.
Hi, I've merged require_expects feature branch into master, and I'm planning to release soon.
I want to list your name as a contributor in README.md and package.json.
Is it okay?
I have no objections. It is your project, after all. ;)
On Sat, Jan 5, 2013 at 12:47 PM, Takuto Wada notifications@github.comwrote:
Hi, I've merged require_expects feature branch into master, and I'm
planning to release soon.
I want to list your name as a contributor in README.md and package.json.
Is it okay?—
Reply to this email directly or view it on GitHubhttps://github.com//issues/6#issuecomment-11917889.
Credited you as contributor. Many many thanks!