Test not reproducible with same seed between rake test and minitest-bisect.

[NicolasWebDev]

Hi,

I am trying to test minitest_bisect.

bundle exec rake test TESTOPTS="--seed 14526"

gives me 5 failures out of 44 runs, while

MTB_VERBOSE=2 bundle exec minitest_bisect -Itest $(find test -type f -name \*_test.rb) --seed 14526

gives me Reproduction run passed? Aborting..
All the tests (44 runs) are run by minitest_bisect, without any error.

Is it because the tests are not run in the same order, because of the find subshell?

I have also tried to reproduce it running

bundle exec ruby -Itest $(find test -type f -name \*_test.rb) --seed=14526

but in this case, only a subset of the tests are run, not all of them (7 out of 44), and there are no failures.

In case it may help:

• rails (4.2.7.1)
• minitest (5.10.1)
• minitest-bisect (1.4.0)

Thanks for your help.

[Ryan Davis]

I assume you're not trying to test minitest_bisect itself and that you're just trying to use minitest_bisect on your project.

Running rake test with -t on should output the command that rake is using to run your tests. You can (sometimes) use that to run via minitest_bisect. If not, then you can add minitest-sprint to your Gemfile, bundle, and then use minitest to reproduce with your $(find ...) bit from above.

To help explain:

bundle exec rake test TESTOPTS="--seed 14526"

runs your tests like normal. Where:

MTB_VERBOSE=2 bundle exec minitest_bisect -Itest $(find test -type f -name \*_test.rb) --seed 14526

doesn't necessarily load everything the same way that rake did. Looking at how rake loads your tests might help.

But:

bundle exec ruby -Itest $(find test -type f -name \*_test.rb) --seed=14526

is nonsensical. It is basically:

ruby -Itest test/first_test.rb test/second_test.rb

which just says "run the first file and the rest of the things are just command-line arguments".

[NicolasWebDev]

Indeed, I am trying to use minitest-bisect on my project, to see how it could help me next time I am faced with random failures.

I have tried rake -t test, rake test -t, rake test TESTOPTS='-v', and neither of these give me the command rake uses to run the tests. The output is:

$ bundle exec rails --version                                                                                                             
Rails 5.0.1                                                                                                                                                                                                       
$ bundle exec rake --version                                                                                                              
rake, version 12.0.0                                                                                                                                                                                              
$ bundle exec rake -t test
** Invoke test (first_time)                                                                                                                                                                                       
** Execute test
Run options: --seed 4660

# Running:

...............F

Failure:
GeneralStoriesTest#test_user_flow [/home/sathors/work/documentation/company/prospection/antonio_bid/7pasospm4r/test/integration/general_stories_test.rb:24]:
Expected: 2
  Actual: 0


bin/rails test test/integration/general_stories_test.rb:10

................

Finished in 0.661467s, 48.3773 runs/s, 57.4481 assertions/s.

32 runs, 38 assertions, 1 failures, 0 errors, 0 skips                                                                                                                                                             
Coverage report generated for MiniTest to /home/sathors/work/documentation/company/prospection/antonio_bid/7pasospm4r/coverage. 262 / 304 LOC (86.18%) covered.  

With a rails 4.2 project (with which I have opened this issue), I seem to have the following bug: rails/rails#24372, so the -t option is not recognized.

I also have tried to install minitest-sprint like you said, but using the same seed, everything passes when I would be expected failures. As you said, if I use the 'find' command, I guess the seed won't be the same than the one used with rake. So I would have to rerun the tests with minitest until I find the same random failures, then take this seed to run it with minitest-bisect???

[Peter Leitzen]

@NicolasWebDev @zenspider

Today, I've stumbled upon the "load order" of the test files when using minitest_bisect in a Rails project.

To make minitest_bisect work on this Rails project I had to use rake's load order via:

ruby -rrake -e 'puts FileList["test/**/*_test.rb"]'

@NicolasWebDev Your example from above would look like:

MTB_VERBOSE=2 bundle exec minitest_bisect -Itest $(ruby -rrake -e 'puts FileList["test/**/*_test.rb"]') --seed 14526

Note that $(find ...) did not work for me. Even $(find ... | sort) gave different results.

I hope this helps 😀

[Peter Leitzen]

@zenspider BTW, after using rake's load order minitest_bisect found the offending test case 👍

Thank you for this awesome gem 💎 💚

[NicolasWebDev]

Well, I am not able to reproduce my test case, because I think that I have deleted the corresponding branch.

Thanks @splattael for your help though, I will refer to it next time!