tpierrain / Diverse

Diverse, the Fuzzer pico library you need to make your .NET tests more Diverse

Diverse

Diverse, the Fuzzer pico library you need to make your .NET tests more Diverse.

use case driven on twitter - ([e-mail])

Diverse added-value

Diverse:

• Provides a set of primitives to easy Fuzz data in your .NET tests in a Diverse manner

• Provides fully randomed values, but every case/test run can be reproduced afterwards in a deterministic manner if needed (e.g.: specific test case, debugging)

• Is a pico library with no external dependency, and is nonetheless compliant with all test frameworks

• Is easily extensible through .NET extension methods over a simple IFuzz interface

Sample

Example of a typical test using Fuzzers. Here, a test for the SignUp process of an API:

[Test]
public void Return_InvalidPhoneNumber_status_when_SignUp_with_an_empty_PhoneNumber()
{
    var fuzzer = new Fuzzer();
            
    // Uses the Fuzzer
    var person = fuzzer.GeneratePerson(); // speed up the creation of someone with random values
    var password = fuzzer.GeneratePassword(); // avoid always using the same hard-coded values
    var invalidPhoneNumber = "";

    // Do your domain stuff
    var signUpRequest = new SignUpRequest(login: person.EMail, password: password, 
                                            firstName: person.FirstName, lastName: person.LastName, 
                                            phoneNumber : invalidPhoneNumber);

    // Here, the quality of the password won't be a blocker for this
    // SignUp process. We just want to check the behaviour with empty phone number
    var signUpResponse = new AccountService().SignUp(signUpRequest);

    // Assert
    Check.That(signUpResponse.Login).IsEqualTo(person.EMail);
    Check.That(signUpResponse.Status).IsEqualTo(SignUpStatus.InvalidPhoneNumber);
}

What are Fuzzers and why should I use them?

Fuzzers are tiny utilities generating data/cases for you and your tests. Instead of hard-coding '[e-mail]' in all your tests (or using 42 as default integer ;-) a Fuzzer will generate any random value for you. Diverse will provide you random but relevant/credible data but may soon provide invalid or unexpected ones too. Thus, you will more easily discover issues in your production code or discover that a test is badly written.

Disclaimer

Diverse is not a Property-based testing framework nor an advanced Fuzzer.

There are debates about what is really a Fuzzer or not. In Wikipedia, one can found the following definition for Fuzzing:

"Fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program."

So far the lib will only provide credible random data and not invalid or unexpected ones.

But it will soon probably do the second part too.

How does Diverse looks like

// First, you can instantiate Fuzzers that will never return the same entries twice (if you want)
var fuzzer = new Fuzzer();

var fuzzerProvidingNoDuplicatedValues = fuzzer.GenerateNoDuplicationFuzzer();

// avoid using always the same hard-coded values in your tests code => use Fuzzers instead
var age = fuzzer.GenerateInteger(minValue: 17, maxValue: 54);

// random but deterministic ;-) Guids
var guid  = fuzzer.GenerateGuid();

// speed up the creation of something relatively 'complicated' and stay *intent driven*
// with random but coherent values (here, the Diverse Person)
var person = fuzzer.GeneratePerson(); 

// or other specific stuffs
var password = fuzzer.GeneratePassword();

// any enum value
var ingredient = fuzzer.GenerateEnum<Ingredient>();

// any entry picked from a list
var candidates = new[] {"one", "two", "three", "four", "five"};
var chosenOne = fuzzer.PickOneFrom(candidates);

// any text, paragraphs, sentences, words in Latin
var text = fuzzer.GenerateText(nbOfParagraphs: 3); // Lorem ipsum...

// or dates
var dateTime = fuzzer.GenerateDateTime();
var dateTimeInRange = fuzzer.GenerateDateTimeBetween("1974/06/08", "2020/11/01");
var otherDateTimeInRange = fuzzer.GenerateDateTimeBetween(new DateTime(1974,6,8), new DateTime(2020, 11, 1));

// or any string following simple patterns
var value = fuzzer.GenerateStringFromPattern("X#A02x"); // e.g.: U3A02k or B6A02t

// or any type actually (either, class, enum)
var player = fuzzer.GenerateInstanceOf<ChessPlayer>();

// Anything you need for your test cases actually
// Diverse Fuzzers are easily extensible ;-)

Every time your test will run, it will use different random values for what matters in your Domain.

The whole idea of a good Fuzzer is to be as easy to use as it is to put an hard-coded value.

Why the name Diverse?

Thanks to ask ;-) Well... It is a matter of fact that the software industry is still not really a super inclusive place right now.

Karens are in minority here ;-)

Indeed, Diverse will help you to make your tests more inclusive and more diverse by picking other things that Karen of John as default first names for instance.

Diverse will help you to quickly generate diverse names or persons from various genders, countries, cultures, etc.

Just pick the primitives you want and check by yourself ;-)

// Examples of Persons created with Diverse:

Mrs. Zahara NGOMA (Female) [e-mail] (married - age: 20 years) 
Ms. Signilde HAUGLAND (Female) [e-mail] (age: 33 years) 
Ms. Imogen WILLIAMS (Female) [e-mail] (age: 41 years) 
Mrs. Zayneb DROGBA (Female) [e-mail] (married - age: 31 years) 
Mr. Samuel MÜLLER (Male) [e-mail] (age: 27 years) 
Mx. Gojko BERNARD (NonBinary) [e-mail] (age: 34 years) 
Ms. Thuong KIM (Female) [e-mail] (age: 75 years) 
Mr. Samuel CONTI (Male) [e-mail] (married - age: 25 years) 
Mrs. Dae BAK (Female) [e-mail] (married - age: 32 years) 
Mx. Okal MENSAH (NonBinary) [e-mail] (married - age: 28 years)

Fully Random, but deterministic when needed! (for debugging)

Use extensible fuzzers that randomize the values for you, but that can be replayed deterministically if any of your tests failed one day (in a specific configuration).

I explained this here in that thread:

see the thread on twitter here

How to deterministically reproduce a test that has failed but only in a very specific case (picked randomly)

1. First, ensure that Diverse's logs will be traced down wherever you want.

All you have to do is to call once the Fuzzer.Log setter:

e.g.: here with NUnit :

    [SetUpFixture]
    public class AllTestFixtures
    {
        [OneTimeSetUp]
        public void Init()
        {
            // We redirect all Diverse's log into NUnit's TestContext outputs
            Fuzzer.Log = TestContext.WriteLine;
        }
    }

2. Consult the report of a failing test and Copy the Seed that was used for it.

Note: Diverse traces the seed used for every test ran. It will look like this:

 ----------------------------------------------------------------------------------------------------------------------
--- Fuzzer ("fuzzer1265") instantiated with the seed (1248680008)
--- from the test: FuzzerWithNumbersShould.GeneratePositiveInteger_with_an_inclusive_upper_bound()
--- Note: you can instantiate another Fuzzer with that very same seed in order to reproduce the exact test conditions
-----------------------------------------------------------------------------------------------------------------------

3. Change your failing test to provide the copied Seed to its fuzzer:

Instead of:

 var fuzzer = new Fuzzer();

calls:

 var fuzzer = new Fuzzer(seed: 1248680008);

That's it! Your test using Diverse fuzzers will be deterministic and always provide the same Fuzzed values.

You can then fix your implementation code to make your test green, or rewrite your badly written test, or keep your test like this so you can have deterministic values in it (nice for documentation).

How to extend Diverse with your own Fuzzing methods

Fortunately, Diverse is extensible. You will be able to add any specific method you want.

All you have to do is to add your own .NET extension methods onto the 'IFuzz' interface

• This will automatically add your own method to any Fuzzer instance

• This will allow your methods to have access to the 'Random' instance of the Fuzzer

  • Interesting to leverage & compose with any other existing Fuzzing function!

  • Mandatory to benefits from the Deterministic capabilities of the library

Example of method extension for IFuzz

The IFuzz interface implemented by all our Fuzzer instances is made to be extended. Let's say that we want to Add a new fuzzing method to dynamically generate an 'Age' instance (part of our specific domain).

All we have to do in our project is to add a new GenerateAVerySpecificAge() extension method that may looks like this:

public static class FuzzerExtensions
{
    public static Age GenerateAVerySpecificAge(this IFuzz fuzzer)
    {
        // Here, we can have access to all the existing methods 
        // exposed by the IFuzz interface
        int years = fuzzer.GeneratePositiveInteger(97);

        // or this one (very useful)
        bool isConfidential = fuzzer.HeadsOrTails();

        // For very specific needs, you have to use the
        // Random property of the Fuzzer
        double aDoubleForInstance = fuzzer.Random.NextDouble();

        return new Age(years, isConfidential);
    }
}

So that we can now see it and call it onto any of the Diverse Fuzzers like this in a test:

[TestFixture]
public class FuzzerThatIsExtensibleShould
{
    [Test]
    public void Be_able_to_have_extension_methods()
    {
        var fuzzer = new Fuzzer(1245650948);

        // we have access to all our extension methods on our Fuzzer
        // (here: GenerateAVerySpecificAge())
        Age age = fuzzer.GenerateAVerySpecificAge();

        Check.That(age.Confidential).IsTrue();
        Check.That(age.Years).IsEqualTo(59);
    }
}

---

"I hope you will enjoy & benefit from adding more and more Fuzzers in your tests, but also to introduce some diversity in your code bases"

Thomas PIERRAIN