bisecting doesn't seem to work

[Aaron Patterson]

I'm not sure if bisecting can work in this case, but here is the issue I'm having.

$ git clone https://github.com/tenderlove/rack.git
$ cd rack
$ git checkout argh
$ ruby -I lib:test  test/spec_directory.rb --seed 5125

This should fail like this:

$ ruby -I lib:test  test/spec_directory.rb --seed 5125
Run options: --seed 5125

# Running:

......FF

Finished in 0.021605s, 370.2804 runs/s, 879.4160 assertions/s.

  1) Failure:
Rack::Directory#test_0003_pass to app if file found [test/spec_directory.rb:42]:
Expected #<Rack::MockResponse:0x007fde5388ec38 @original_headers={"Content-Type"=>"text/plain", "Content-Length"=>"28", "X-Cascade"=>"pass"}, @errors="", @body_string=nil, @status=404, @header={"Content-Type"=>"text/plain", "Content-Length"=>"28", "X-Cascade"=>"pass"}, @writer=#<Proc:0x007fde5388e5a8@/private/tmp/rack/lib/rack/response.rb:32 (lambda)>, @block=nil, @length=28, @body=["Entity not found: /cgi/test\n"]> to be ok?.


  2) Failure:
Rack::Directory#test_0007_uri escape path parts [test/spec_directory.rb:84]:
Expected #<Rack::MockResponse:0x007fde5386cf48 @original_headers={"Content-Type"=>"text/plain", "Content-Length"=>"38", "X-Cascade"=>"pass"}, @errors="", @body_string=nil, @status=404, @header={"Content-Type"=>"text/plain", "Content-Length"=>"38", "X-Cascade"=>"pass"}, @writer=#<Proc:0x007fde5386ca98@/private/tmp/rack/lib/rack/response.rb:32 (lambda)>, @block=nil, @length=38, @body=["Entity not found: /cgi/test+directory\n"]> to be ok?.

8 runs, 19 assertions, 2 failures, 0 errors, 0 skips

If I run with minitest bisect, it takes too long, so I just kill it:

$ MTB_VERBOSE=2 minitest_bisect -Ilib:test --seed 5125 test/spec_directory.rb

I expect minitest bisect to figure out that the test causing the issue is the test I added, which is breaking because it overwrites the app value from the closure. Here is the closure value, here is where it gets overwritten.

[Aaron Patterson]

If you run with this seed, they will all pass: --seed 52007

[Ryan Davis]

Cute repro! I have it on my side. The culprit count goes up to 64 even tho there's like, 8 tests... something hinky.

[Ryan Davis]

Non-rack Repro:

require 'minitest/autorun'

describe "bug 8" do
  app = 42

  it 'serves directories with + in the name' do
    app = 24

    assert_equal 24, app
  end

  it "serve directory indices" do
    assert_equal 42, app
  end

  it "pass to app if file found" do
    assert_equal 42, app
  end

  it "serve uri with URL encoded filenames" do
    assert_equal 42, app
  end

  it "not allow directory traversal" do
    assert_equal 42, app
  end

  it "404 if it can't find the file" do
    assert_equal 42, app
  end

  it "uri escape path parts" do # #265, properly escape file names
    assert_equal 42, app
  end

  it "correctly escape script name" do
    assert_equal 42, app
  end
end

[Ryan Davis]

Got it:

% ruby -I../lib ../bin/minitest_bisect -Ilib:test --seed 5125 test/spec_directory.rb
reproducing...
reproduced
# of culprit methods: 4
# of culprit methods: 2
# of culprit methods: 1
# of culprit methods: 1

Minimal methods found in 4 steps:

/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/bin/ruby -Ilib:test -e 'require "./test/spec_directory.rb"' -- --seed 5125 --server 18349 -n "/^(?:Rack::Directory#(?:test_0001_serves directories with \+ in the name|test_0003_pass to app if file found|test_0007_uri escape path parts))$/"

Final reproduction:

Run options: --seed 5125 -n "/^(?:Rack::Directory#(?:test_0001_serves directories with \\+ in the name|test_0003_pass to app if file found|test_0007_uri escape path parts))$/"

# Running:

.FF

Finished in 0.005751s, 521.6484 runs/s, 521.6484 assertions/s.

  1) Failure:
Rack::Directory#test_0003_pass to app if file found [/Users/ryan/Work/p4/zss/src/minitest-bisect/dev/rack/test/spec_directory.rb:42]:
Expected #<Rack::MockResponse:0x007f84a3038b60 @original_headers={"Content-Type"=>"text/plain", "Content-Length"=>"28", "X-Cascade"=>"pass"}, @errors="", @body_string=nil, @status=404, @header={"Content-Type"=>"text/plain", "Content-Length"=>"28", "X-Cascade"=>"pass"}, @writer=#<Proc:0x007f84a3038908@/Users/ryan/Work/p4/zss/src/minitest-bisect/dev/rack/lib/rack/response.rb:32 (lambda)>, @block=nil, @length=28, @body=["Entity not found: /cgi/test\n"]> to be ok?.


  2) Failure:
Rack::Directory#test_0007_uri escape path parts [/Users/ryan/Work/p4/zss/src/minitest-bisect/dev/rack/test/spec_directory.rb:84]:
Expected #<Rack::MockResponse:0x007f84a3040130 @original_headers={"Content-Type"=>"text/plain", "Content-Length"=>"38", "X-Cascade"=>"pass"}, @errors="", @body_string=nil, @status=404, @header={"Content-Type"=>"text/plain", "Content-Length"=>"38", "X-Cascade"=>"pass"}, @writer=#<Proc:0x007f84a3047e80@/Users/ryan/Work/p4/zss/src/minitest-bisect/dev/rack/lib/rack/response.rb:32 (lambda)>, @block=nil, @length=38, @body=["Entity not found: /cgi/test+directory\n"]> to be ok?.

3 runs, 3 assertions, 2 failures, 0 errors, 0 skips

[Ryan Davis]

Fixed and will be released soon.