"Doubling up" of tests - Old results hanging around
Daniel15 opened this issue · comments
This is an odd issue to explain, but I'll try to do it as best I can.
Whenever I change my tests, JsTestDriver seems to "cache" the old tests, and combine the old results with the new results. As an example, one of my spec files has 7 tests in it. This works fine the first time I run JsTestDriver, and also works fine on a second run without changing anything. However, if I change my spec file (even insignificantly - ie. adding whitespace) and run the tests again, JsTestDriver now says I have 14 tests!
If I change a test to fail intentionally and then change it back so it passes, the fail remains and appears on every single test run, until I restart the JsTestDriver server.
I'm using Jasmine 1.1.0, JsTestDriver 1.1.3d, Prototype 1.7, and the Jasmine Prototype adapter.
This was my fault - I wasn't using the provided shell scripts, and forgot to pass the --reset parameter to the JsTestDriver client. Once I did this, it started working fine.
have you tried --reset?
On Tue, Dec 20, 2011 at 4:38 AM, Daniel <
reply@reply.github.com
wrote:
This is an odd issue to explain, but I'll try to do it as best I can.
Whenever I change my tests, JsTestDriver seems to "cache" the old tests,
and combine the old results with the new results. As an example, one of my
spec files has 7 tests in it. This works fine the first time I run
JsTestDriver, and also works fine on a second run without changing
anything. However, if I change my spec file (even insignificantly - ie.
adding whitespace) and run the tests again, JsTestDriver now says I have 14
tests!If I change a test to fail intentionally and then change it back so it
passes, the fail remains and appears on every single test run, until I
restart the JsTestDriver server.
Reply to this email directly or view it on GitHub:
#20
Hrm... I am not sure that this is a solution so much as a workaround. I imagine that the --reset switch would cause all scripts and tests to be reloaded in all slaves captured by the server. While it works, it could be faster, and eliminates one of the benefits of using js test driver.
Looking at the code, I am sure @ibolmo understands that its a problem with the top level calls to describe being intercepted and remembered in a global variable which never has re-loaded describes cleaned up. I spent some time pouring through js test driver to see if there was a good hook to do this, and it seems the best thing is the js test driver test case. All test cases for a certain script file are removed every time that script file is loaded.
I've been thinking that on intercept, it may be possible to wrap and delay a describe call with a test case. Though this gets complicated/unhelpful when thinking about runner-level before/afterEach calls... since those have to be associated with every describe/testcase that is created for the file. I am guessing that's the main reason this solution wasn't considered?
Well, its not as graceful,but I am wondering if additional, top-level api call can't be added to the bridge to augment the jasmine api? Something like jasminefileis('some_unique_id'). This unique id can then be associated with each intercepted function and on each call associated functions are removed from the global caches?
I think I can justify spending some time on this if it would be accepted. Though I prefer the elegant solution, I would pursue the second option if the first one isn't technically feasible.
An elegant solution would be great. Using --reset does work but it makes testing in IE6 considerably slower (I'm guessing due to reloading everything). This is one of the main reasons I'm using JSTestDriver - Testing on multiple IE versions simultaneously.
Well, I think that an elegant solution is possible. I've been discussing it on the jstd list:
https://groups.google.com/d/topic/js-test-driver/lENI9Wy5mFE/discussion
I believe I am going to try to implement it and request a pull in the next few days.
This would be great. Thank you for the help; keep us posted.
On Wed, Feb 8, 2012 at 2:04 PM, Steven <
reply@reply.github.com
wrote:
Well, I think that an elegant solution is possible. I've been discussing
it on the jstd list:
https://groups.google.com/d/topic/js-test-driver/lENI9Wy5mFE/discussionI believe I am going to try to implement it and request a pull in the next
few days.
Reply to this email directly or view it on GitHub:
NP. It was fairly straight-forward once I got rid of all my defects. I am going to wait from the jstd team to make sure the approach is sound, and contact my co.'s legal department or something in the meantime to make sure I am following our internal process for contributing to OSS projects.
Hopefully this won't take too long.
Hrm... it doesn't quite work the way it should.
It seems that the first set of intercepts after server start are always kept around after N reloads of the file but you would at least only have the first set and the latest set. Because the loadSource on the plugin is not called for the first set of sources to load :(
If you --reset the test runner, then you only keep the latest set of intercepts.... so a temporary way to deal with the issue would be to immediately run the first set of test with "--reset" (I think... I haven't actually tried this yet)...
If I get a chance I may have to look in to jstd to see if there is a way to fix this... but contributing there may be a bit more difficult for me.
OK. Available as pull request #21. Please have a look.
I think this is a valid issue. Let say you have a test case like:
MyTestCase = TestCase("js-test-driver-testcase");
MyTestCase.prototype.testA = function(){
};
describe("jasmine-testcase", function () {
it("assert all good", function () {
});
});
Now if you execute the test and save the testcase file and rerun the test again you will see that js-test-driver test case is reported single time but jasmine based testcase is reported twice. I bumped into this while resolving NetBeans issue http://netbeans.org/bugzilla/show_bug.cgi?id=225381 Unfortunately in NetBeans case I cannot use "--reset" workaround because js-test-driver is executed via its Java API and Java API does not allow to set reset flag.