Compatibility issue with XUnit's `IClassFixture`

[Urho Niemelä]

Describe the bug
I'm using XUnit's IClassFixture to share context between tests in the same class. First, one test sets the return value of a substitute method with Returns(). Later during test execution, another test sets the method's return value to something else. However, The later test also uses the return value that was set in the earlier test.

To Reproduce
Let's say I have a NumberChecker app that checks numbers. CheckIfNumberExists first validates the given number, and on successful validation, checks if it exists.

    public interface INumberChecker
    {
        public bool Validate(int num);
        public bool Exists(int num);
    }

    public class MyApp
    {
        private readonly INumberChecker _numberChecker;
        public MyApp(INumberChecker myInterfaceImpl)
        {
            _numberChecker = myInterfaceImpl;
        }
        public void CheckIfNumberExists(int num)
        {
            if (_numberChecker.Validate(num))
            {
                _numberChecker.Exists(num);
            }
        }
    }

For my tests, I create a Fixture that should be shared with all tests. The fixture holds a substitute for INumberChecker called SharedSubstitute, and initialises a new MyApp:

    public class Fixture
    {
        public readonly INumberChecker SharedSubstitute;
        public readonly MyApp App;
        public Fixture()
        {
            SharedSubstitute = Substitute.For<INumberChecker>();
            App = new MyApp(SharedSubstitute);
        }
    }   

I want to test two things:

1. if number is valid, the Exists() method is called
2. if number is not valid, the Exists() should not be called.
   My Tests class inherits XUnit's IClassFixture<Fixture> so that Fixture will be shared between all the tests.

 public class Tests : IClassFixture<Fixture>
    {
        private readonly Fixture _fixture;

        public Tests(Fixture fixture)
        {
            _fixture = fixture;
        }

       //1.
        [Fact]
        public void ExistsIsCalledWhenValidationIsSuccessful()
        {
            //first we make sure that Validate() passes so that Exists() gets called
            _fixture.SharedSubstitute.Validate(Arg.Any<int>()).Returns(true);

            //act:
            _fixture.App.CheckIfNumberExists(1);

            //then we test that the substitute received a call to Exists() 
            _fixture.SharedSubstitute.Received().Exists(Arg.Any<int>());
        }
        
        //2.
        [Fact]
        public void ExistsIsNotCalledWhenValidationFails()
        {
            //now we pretend that Validate() fails, i.e. returns false.
            _fixture.SharedSubstitute.Validate(Arg.Any<int>()).Returns(false);

            //act:
            _fixture.App.CheckIfNumberExists(1);

            //we shouldn't expect a call to Exists()
            _fixture.SharedSubstitute.DidNotReceive().Exists(Arg.Any<int>());
        }
    }

Expected behaviour
I would expect that the second test sets the return value of Validate to false and the test passes. However, when running both tests togther (so that they share the Fixture), the second test fails, because it thinks Validate should still return true.

Environment:

• NSubstitute version: [5.1.0]
• NSubstitute.Analyzers version: [CSharp 1.0.16]
• Platform: [dotnet 6.0 on Windows, XUnit]

Additional context

• if you run the tests individually, the problem won't occur, because both runs will get a fresh context.
• You should get the error, when you run both tests together (for example with dotnet test)
• If you remove the IClassFixture inheritance and initialise everything in the Tests constructor, the problem goes away.

[GeraldLx]

That has nothing to do with ClassFixture or NSubstitute.

Your test is just wrong.

By using the ClassFixture you have only one instance of your substitute for both tests.

So your first tests sets up the substitute´s Validate function to return true AND calls CheckIfNumberExists. This of course leads to ca call of Exists.

Then your second test runs. You change the Validate to return false AND call CheckIfNumerExists a second time. This of course leads to no call of Exists.

But then you check the number of calls to Exists. Of course the number of calls is 1 if you run both tests. But you expect it to be zero. Therefore the 2nd test fails.

Just change your test like this:

public class Tests
{
  private readonly Fixture _fixture;

  public Tests()
  {
    _fixture = new Fixture();
  }

  //1.
  [Fact]
  public void ExistsIsCalledWhenValidationIsSuccessful()
  {
    //first we make sure that Validate() passes so that Exists() gets called
    _fixture.SharedSubstitute
      .Validate(Arg.Any<int>())
      .Returns(true);

    //act:
    _fixture.App.CheckIfNumberExists(1);

    //then we test that the substitute received a call to Exists() 
    _fixture.SharedSubstitute.Received().Exists(Arg.Any<int>());
  }

  //2.
  [Fact]
  public void ExistsIsNotCalledWhenValidationFails()
  {
    //now we pretend that Validate() fails, i.e. returns false.
    _fixture.SharedSubstitute
      .Validate(Arg.Any<int>())
      .Returns(false);

    //act:
    _fixture.App.CheckIfNumberExists(1);

    //we shouldn't expect a call to Exists()
    _fixture.SharedSubstitute.DidNotReceive().Exists(Arg.Any<int>());
  }
}

and it works, regardless of running both or only one test.

But what you experienced in your version is just by design

[Urho Niemelä]

That makes total sense, looks like I failed to consider that. Thanks for the response!