Corvus.Testing

This provides a library of useful testing extensions, primarily focussed on SpecFlow operations.

It is built for netstandard2.0.

The SpecFlow specific libraries contain additional bindings; to use these, you will need to add a specflow.json file to any project wishing to use them. Add entries to the stepAssemblies array for each of the Corvus libraries you need to use:

{
  "stepAssemblies": [
    { "assembly": "Corvus.Testing.SpecFlow" },
    { "assembly": "Corvus.Testing.AzureFunctions.SpecFlow" },
  ]
}

Features

This library offers the following features.

Container Bindings (Corvus.Testing.SpecFlow)

Corvus.Testing.SpecFlow can simplify the use of dependency injection. It can automatically create a dependency injection service collection (an instance of the IServiceCollection type defined by the Microsoft.Extensions.DependencyInjection.Abstractions component). It offers your tests an opportunity to populate this collection during the early phases of test execution. It then automatically builds an IServiceProvider from the service collection, and makes this accessible to your test. It disposes the provider at the end, ensuring any dependencies that need to clean up will have their Dispose methods called.

This feature offers two modes: you can either arrange for the service collection to be created, built, and torn down for individual scenarios, or you can create one that is created at the start of a feature, in which the service provider is shared by all scenarios within the feature, and the services are disposed only after all scenarios have run.

In most cases per-scenario mode is better, because it improves isolation between tests. However, if service initialization is particularly slow, feature-level containers might offer a useful escape hatch.

To use per-scenario mode, use the @perScenarioContainer tag. You can either put this on each of the individual scenarios you need, or put it at the top of the feature file, in which case every scenario in that feature will get its own container. To use per-feature mode, put a @perFeatureContainer tag at the top of the feature file. (You cannot use both modes at once.)

To populate the service collection you must write a binding that runs at the appropriate time. For per-scenario collections, it will look like this:

[BeforeScenario("@perScenarioContainer", Order = ContainerBeforeScenarioOrder.PopulateServiceCollection)]
public static void PopulateScenarioServiceCollection(ScenarioContext scenarioContext)
{
    ContainerBindings.ConfigureServices(scenarioContext, services =>
        services.AddLogging()
            .AddSingleton<IStore, FakeStore>());
}

and per-feature collections work in almost exactly the same way, largely just changing "Scenario" to "Feature":

[BeforeFeature("@perFeatureContainer", Order = ContainerBeforeFeatureOrder.PopulateServiceCollection)]
public static void PopulateFeatureServiceCollection(FeatureContext featureContext)
{
    ContainerBindings.ConfigureServices(featureContext, services =>
        services.AddLogging()
            .AddSingleton<IStore, FakeStore>());
}

The method name doesn't matter. It's the attribute that's significant. Note that by specifying the same @perScenarioContainer (or @perFeatureContainer) tag that Corvus.Testing.SpecFlow is looking for, this binding will run for any tests that specify that tag. If you want to use this container feature in multiple tests but you want to use different service configuration in different tests, you can define your own tag, e.g. your feature file might start:

@perScenarioContainer
@adminTestContainerInitialization

And then you would specify that more specialized @adminTestContainerInitialization tag in your bindings instead of @perScenarioContainer.

To obtain services from the DI container, you can write code like this:

IServiceProvider serviceProvider = ContainerBindings.GetServiceProvider(this.scenarioContext);
MyService svc = this.ServiceProvider.GetRequiredService<MyService>()

The ContainerBindings.GetServiceProvider method is overloaded, accepting either a ScenarioContext or a FeatureContext. If you used the @perScenarioContainer tag you should pass the scenario context as the example above does, but if you're using @perFeatureContainer pass the feature context instead.

GetServiceProvider will work from inside any step, whether it's Given, When, or Then. But if you need to write a custom Before... binding that has access to the service provider, you'll need to make sure it runs at the appropriate moment. Since Corvus.Testing.SpecFlow relies on this binding mechanism to create and initialize the services, you need to make sure that your code runs at the right moment, which means using the Order property on these SpecFlow binding attributes. The bindings shown above that populate the IServiceCollection set their Order property with constants supplied by Corvus.Testing.SpecFlow. This ensures that these bindings run after Corvus.Testing.SpecFlow has created the service collection but before it has built the provider. If you want to access services from DI you will need to run after the provider has been built, which you can do by specifying a different constant for the Order:

[BeforeScenario("@adminTestContainerInitialization", Order = ContainerBeforeScenarioOrder.ServiceProviderAvailable)]
public static void ServiceProviderAvailableBeforeScenario(ScenarioContext scenarioContext)
{
    IServiceProvider serviceProvider = ContainerBindings.GetServiceProvider(scenarioContext);
    MyService svc = this.ServiceProvider.GetRequiredService<MyService>()
    // etc.
}

(This presumes that we've got some test-specific work going on. If you wanted this binding to run any time you used a per-scenario container you'd put @perFeatureContainer instead of @adminTestContainerInitialization.)

Note that the Order constants that Corvus.Testing.SpecFlow provides space to allow you to control the order of multiple bindings of your own if necessary. The BuildServiceProvider is 9999 higher than PopulateServiceCollection, so if you need to perform multiple bindings in between the the service collection being created, and the final service provider being built from that collection, you can do so, and you can control their ordering. (E.g., you can have one binding with Order = ContainerBeforeScenarioOrder.PopulateServiceCollection, and then another with ContainerBeforeScenarioOrder.PopulateServiceCollection + 1.)

Exceptions during container disposal are handled using the Teardown Exception Handling mechanism described below, meaning that if errors occur, they will not prevent other teardown from happening, but will still eventually be reported. (Although be aware that depending on how you run your tests, feature-level errors will not necessarily be reported as errors, due to limitations inherent in how .NET test runners integrate with build and development tools.)

Child Object Value Retriever (Corvus.Testing.SpecFlow)

If a scenario or feature specifies the @useChildObjects tag, this registers a Value Retrieved with SpecFlow enabling named objects in the scenario context to be referred to with a {name} syntax. So if an earlier stage of a test puts something into the scenario context with the key transactionId you could write {transactionId} in a SpecFlow feature fle to retrieve that value from the context and pass it in as the argument to a step.

Azure Functions Launch (Corvus.Testing.AzureFunctions and Corvus.Testing.AzureFunctions.SpecFlow)

The Corvus.Testing.AzureFunctions library contains a FunctionsController class that can be used to run Azure Functions as separate processes as part of a test.

For usage examples, see the Corvus.Testing.AzureFunctions.Xunit.Demo and Corvus.Testing.AzureFunctions.SpecFlow.Demo, which show the various ways these classes can be used.

SpecFlow-specific bindings can be found in Corvus.Testing.AzureFunctions.SpecFlow, which defines a Given step that matches this pattern:

I start a functions instance for the local project '(.*)' on port (.*)

This enables you to write a test step such as:

    Given I start a functions instance for the local project 'WorkflowFunction.Host' on port 8881

The effect of this will be to launch the local function emulator with the WorkflowFunction.Host project, telling it to listen on port 8881. It will wait until the function's standard output prints the message indicating that it is ready to accept incoming requests.

You can launch multiple functions in a single scenario as long as they are all listening on different ports. This can be helpful for integration tests.

Corvus.Testing.AzureFunctions.SpecFlow includes an unconditional [AfterScenario] binding that will run after every single test, and it checks to see if any functions were launched, and if so, shuts it down and waits until the process exits. (It waits to avoid port conflicts if another test wants to go on to run a function on the same port.) It also collects all standard output including any errors, and prints this to the test process's console. This can be useful for diagnosing failures, because relevant information is often written to these channels. (This library does not run the function in a console window, so this is the only way to see the output. A console window would not be a good fit for tests running as part of an automated build in any case.)

You will need the Functions development tools to be installed for this to work. If you don't have that, the test will fail, reporting the npm command you need to run to fix it. If you're using Azure DevOps pipelines, there's a task for this.

Teardown Exception Handling (Corvus.Testing.SpecFlow)

If you do any non-trivial work during After... bindings (e.g., tearing down a function) it is often important to ensure that all the bindings run. This is problematic in cases where failures might occur at this stage, because the only way a binding can report failure is by throwing an exception, and if it does so, it will normally prevent all other bindings from running, because it causes the test to come to an abrupt halt. (SpecFlow doesn't have a concept of "fail, but continue to clean up".) This is especially problematic for integration tests that create external resources. (E.g., if your test creates resources in Azure, it can be costly if you fail to clean these up when you're done.)

Corvus.Testing.SpecFlow provides a mechanism by which you can run code in some sort of After... binding, and be free to throw exceptions without that halting all further cleanup, while still eventually seeing the error. You do this through a helper like this:

[AfterScenario]
public void TeardownFunctionsAfterScenario()
{
    this.scenarioContext.RunAndStoreExceptions(this.functionsController.TeardownFunctions);
}

The RunAndStoreExceptions method here is an extension method on ScenarioContext. There's also one for FeatureContext.

This is the code in Corvus.Testing.AzureFunctions.SpecFlow that tears down any functions created with the Azure Functions Launch feature described earlier. The code that performs this work is called TeardownFunctions and it's an instance member of the FunctionsController class, and it has been passed here as a delegate argument to RunAndStoreExceptions.

RunAndStoreExceptions lets you throw exceptions without risk. It catches any exceptions that emerge from your code, stores them, enables all other teardown to proceed, and then at the very last moment, it checks to see if any exceptions were detected this way, and if they were, it reports them all in one go by throwing an AggregateException containing every failure.

To work, Corvus.Testing.SpecFlow defines unconditional AfterScenario and AfterFeature bindings each with an Order of int.MaxValue. This means they will only truly be the last thing to run for the scenario or feature if nothing else tries the same thing. If you want to use this feature, you will need to ensure that you're not using anything else that also depends on being able to be the very last thing that happens.

Running the Tests

Some of the tests require NodeJS and the Functions development tools to be installed. If you don't have that the test will fail, reporting either that npm was not found or the npm command you need to run to fix it.

choco install nodejs
npm install -g azure-functions-core-tools@3

NOTE: You may need to reboot to get Visual Studio to 'see' the NodeJS installation.

The tests can be run from the command-line using:

dotnet build .\Solutions\Corvus.Testing.sln -c Debug
dotnet test .\Solutions\Corvus.Testing.sln -c Debug

Licenses

Corvus.Testing is available under the Apache 2.0 open source license.

For any licensing questions, please email licensing@endjin.com

Project Sponsor

This project is sponsored by endjin, a UK based Microsoft Gold Partner for Cloud Platform, Data Platform, Data Analytics, DevOps, and a Power BI Partner.

For more information about our products and services, or for commercial support of this project, please contact us.

We produce two free weekly newsletters; Azure Weekly for all things about the Microsoft Azure Platform, and Power BI Weekly.

Keep up with everything that's going on at endjin via our blog, follow us on Twitter, or LinkedIn.

Our other Open Source projects can be found on GitHub

Code of conduct

This project has adopted a code of conduct adapted from the Contributor Covenant to clarify expected behavior in our community. This code of conduct has been adopted by many other projects. For more information see the Code of Conduct FAQ or contact hello@endjin.com with any additional questions or comments.