Breaks order setting for Rails and Minitest

[Chris Nicola]

No matter what you set test_order to on either MiniTest::Test or ActiveSupport::TestCase the tests will always run out of order with this gem.

I've created a simple example of the problem here: https://github.com/chrisnicola/minitest_testing

Just run the tests with TESTOPTS="-v" rake to see the order change for the tests that rely on this gem and that they do not change for tests which only rely on minitest/spec. I've reviewed the code for this Gem but I can't figure out why this happens.

[Ken Collins]

When using a outer describe block that does not resolve the described class to a constant, you are indeed in raw MiniTest::Spec subclass and not a descendant of ActiveSupport::TestCase. I highly recommend you stick with the outer Rails subclassing.

So, in order to get results you want... you have to use the Rails' interface. Because this project does aim to work in that framework. To do that, open up your config/environments/test.rb and set the following. The default in that file should be set to :random.

config.active_support.test_order = :alpha

So I set it to :alpha and ran the tests again. And things appear to be working great.

class MiniTestSpecRailsTest < ActiveSupport::TestCase
  it('a') { assert true }
  it('b') { assert true }
  it('c') { assert true }
  it('d') { assert true }
  describe 'whatever' do
    it('a') { assert true }
    it('b') { assert true }
    it('c') { assert true }
    it('d') { assert true }
  end
end

$ ./bin/rake test TESTOPTS='-v'
MiniTestSpecRailsTest::whatever#test_0001_a = 0.00 s = .
MiniTestSpecRailsTest::whatever#test_0002_b = 0.00 s = .
MiniTestSpecRailsTest::whatever#test_0003_c = 0.00 s = .
MiniTestSpecRailsTest::whatever#test_0004_d = 0.00 s = .
MiniTestSpecRailsTest#test_0001_a = 0.00 s = .
MiniTestSpecRailsTest#test_0002_b = 0.00 s = .
MiniTestSpecRailsTest#test_0003_c = 0.00 s = .
MiniTestSpecRailsTest#test_0004_d = 0.00 s = .

Here is the source excerpt from the AcitveSupport::TestCase source. Hope that helps!

# Sets the order in which test cases are run.
#
#   ActiveSupport::TestCase.test_order = :random # => :random
#
# Valid values are:
# * +:random+   (to run tests in random order)
# * +:parallel+ (to run tests in parallel)
# * +:sorted+   (to run tests alphabetically by method name)
# * +:alpha+    (equivalent to +:sorted+)
def test_order=(new_order)
  ActiveSupport.test_order = new_order
end

[Chris Nicola]

Ah sorry that's my fault. Rails 4.2.6 is setting it to :random in test.rb when generating a new project. I understood :sorted was the default until Rails 5.0 (as per their docs). I guess it is the "default" which is then overridden by the generator.

This means I still have an undiagnosed issue on my actual project's test order. Sigh, thanks for the help.

[Chris Nicola]

@metaskills ok I dug a bit deeper. The tests are in fixed order, but the outer describes are not. With multiple describe blocks the blocks of tests are output in random order, while the inner tests are fixed.

[Ken Collins]

Right... cause the describes are a new class and scope. The block it self would be randomized in relation to that scope. For example... MiniTestSpecRailsTest::whatever#test_#{n} in my example above could be before or after the MiniTestSpecRailsTest#test_#{n} group even when :alpha is run. I think this is the core of your issue? Seems to be the default behavior of MiniTest too.

[Chris Nicola]

@metaskills yeah I think it is. I'll talk to Ryan and see if it can be fixed. I'm not a big fan of the tests running in random order. Or at least the output being randomly ordered.

[Ken Collins]

Cool! Going to close this issue. If you think we need to re-open it and/or do something, please feel free to reply and let me know.

[Chris Nicola]

No, not much you can do sadly since Ryan decided that this was desired behaviour.