before/setup inconsistency
jimmykarily opened this issue · comments
I noticed an inconsistency between setup method and before blocks. I always considered them to be equivalent but here is what happens:
This gem defines the before method here.
The setup method called is this one which according to the comment:
# Add a callback, which runs before <tt>TestCase#setup</tt>.
will call the code in the before block before TestCase#setup.
This creates the following problem (real scenario follows)
I define setup in my test_helper file and I run DatabaseCleaner.start in there. This is supposed to run before any fixtures are added in the database. I create fixtures with FactoryGirl inside before blocks in tests. Those fixtures won't get cleaned because they where created before my setup method gets to run
(end of real scenario)
The actual problem is that if I call before do instead of def setup in my test_helper.rb file then the order is reversed. The code in helper's before runs first then the code in test's before block (and the real scenario bug is gone). I think many people consider setup and before to be equivalent and I've seen the same "bug" in other projects going unnoticed. If this is by design then I think is should be better documented.
I created a demo app to demonstrate the problem here. Simply pull the repo and call bin/rake test test/models/dummy_model_test.rb and see the problem live.
P.S.
There is an other implementation of before in minitest/spec which seems to be doing it correctly. Not sure if this is the one overriden though.
@jimmykarily Sorry for the lag... reading this now.
OK, some backstory... this gem aims to integrate with Rails and eventually dissolve once rails-core allows the DSL spec in ActiveSupport::TestCase. I can't remember how mintiest-rails does it... but we build on to of ActiveSupport's test case and specifically their usage of callback chains for setup & teardown. I mention that here in the docs.
When you say you "define setup" are you defining a method? If so, I am not sure that works without calling super. Even tho [the guides] talk about it, I think defining a method is not the way to go. Also, your statement about running before fixtures if false. Test setups run before each test which is all after the DB test prepare and fixture load. Trust me, this is how Rails works. You can see by removing this gem and changing before do to setup do, the result should be the same.
May I suggest that you approach your problem by not introducing the problem. I am a huge advocate of Rails defaults... and that means transactional fixtures. I happen to think YAML sucks for fixture and I like to use Factories, good factories that leverage your models. To that end, I created the named_seeds gem. The idea is to
- Allow Rails to maintain the test DB. Clone structure etc.
- Allow Rails to use transactional fixtures (really transactions). Fixtures optional.
- Use Factories to augment or replace fixtures. NamedSeeds ensure they load at right time.
- Run each test with transactions... do not clean the DB for each test.
@metaskills please give the demo app a shot. You can see in the test_helper both a before do call and a def setup. If you run bin/rake test dummy_model_test.rb with only before do in test_helper and then only with def setup you will see that the order of the puts calls is not the same. In the first case the before block in the test file is run after the before block in the test helper. In the second case, the before block in the test file is run before the setup method in the test_helper.
I suppose the fixtures you refer to ("Also, your statement about running before fixtures if false. Test setups run before each test which is all after the DB test prepare and fixture load") are those defined in ymls (the fixtures Rails understands and creates before anything else). The fixtures that create my problem are FactoryGirl fixtures created in before blocks inside the test files themselves. I don't use yml fixtures at all because they don't play well with my Capybara/Poltergeist setup (pretty much what this article describes). I will check named_seeds but I think the problem is inherent in the way transactions work so it might not be a solution to this problem.
In any case, I don't think this is a serious bug but it can go unnoticed for long (You only see the relics in the test database when they actually create a problem in other tests). Of course that is, only if you are creating fixtures like I do but I don't think this is a rare pattern.
OK... checking this out now... stay tuned.
OK... this took me awhile because I got side tracked trying to figure out why rake test did not work. Seems it was due to the require not being at the top of the test file. I get side tracked like that often ;) just wanted to make sure that was not a cause of some other symptom.
Let me start off by posting some output from the app to talk around. You can see here that I added names to each put statement for clarity.
With minitest-spec-rails gem.
Sans that janky setup method definition. This looks right.
ActiveSupport::TestCase #before
DummyModelTest - #before
ActiveSupport::TestCase - #setup
Changed before blocks to setup do blocks.
By doing this I am directly hooking into ActiveSupport's setup/teardown life cycle. Doing this also sets up the next test to remove the minitest-spec-rails gem. Again, this looks good sans that janky setup def method.
ActiveSupport::TestCase #before (as setup block)
DummyModelTest - #before (as setup block)
ActiveSupport::TestCase - #setup
WithOUT minitest-spec-rails gem.
Also note that I changed the it method to a test one. Again this looks perfectly normal.
ActiveSupport::TestCase #before (as setup block)
DummyModelTest - #before (as setup block)
ActiveSupport::TestCase - #setup
In Conclusion
I am not seeing a problem here other than we should not be using that old def setup method. I am going to make a pull request to Rails to change the documentation on that because I am sure it is not supported.
@jimmykarily What am I missing? Is your issue with that def setup usage? If so, don't use it... if not, ping me back and point me to the right place. I really want to make sure this gem works for you. Most important, that I am not being dense and missing your points. Cheers man.
Also, your statement about running before fixtures if false. Test setups run before each test which is all after the DB test prepare and fixture load
Agreed on that I'd have to back read to see where I got that wrong. But that was certainly what I was trying to convey too.
@metaskills first of all, thank you for your patience and time :). Regarding the require 'test_helper' missing line, I must have accidentally removed that (it is needed indeed).
Is setup method really not to be used anymore? I can still see before and setup being referenced in documentation as interchangeable methods (E.g. "If you want to extend your test using setup/teardown via a module, just make sure you ALWAYS call super. before/after automatically call super for you, so make sure you don't do it twice.). If that is the case maybe we should ping people to change the docs or at least highlight the difference between them. I already solved the issue as you suggest, I use before everywhere and I'm done with it. I'm just trying to understand whether this is a bug or bad documentation (or maybe I just missed some part of it). Whichever the case we should better fix it to avoid other people facing the same issue.
I can still see before and setup being referenced in documentation as interchangeable methods
Well, thats true and not true... let me explain.
You typically write setup/teardown in Minitest when you are using the original Test::Unit style now called Minitest::Test. You do not use these setup methods in the Minitest::Spec style subclass. Note, Minitest::Spec is on top of Minitest::Test, so you technically could. Of course, this is complicated by...
Rails base test case ActiveSupport::TestCase is not a simple subclass of the Minitest::Test class. Rails has a long history, and with that comes some behavior that have predated things like Rspec, Shoulda and Minitest's modern behavior. The most notable of these is what we call the setup and teardown behavior which allowed Rails users to do multiple callback style chains for their tests without having to implement ad hoc module patterns, mixing things in, and calling super all the time. This also allows very compassable test cases by stacking modules in. Rails makes great use of these in each of their major base test cases, for example ActionController::TestCase.
So, this all comes to a head when you mix the two. Rails is now on top of Minitest, specifically Minitest::Test, but with that, you get Rails conventions that they themselves follow. Writing basic setup/teardown without the block syntax is going to bypass the callback chain which as you have seen, gets first shot at running due to how ActiveSupport is implemented. This gem aims to integrate with Rails philosophies and as such actually does very little. We even alias before/after to that same callback chain. This is a big deal IMO because I think your test suite should be coupled to Rails so that your tests do not change much as you upgrade.
In closing, I see a few things I can bring up with Rails core... When I do, I will link here. All cool?
Super cool @metaskills. You are extra awesome!
Thanks! Closing this issue. It really helps to talk thru this stuff so others can find it too. Cheers!