Create fluent-assertions library to improve consumer test readability

Question

Create fluent-assertions library to improve consumer test readability

sgmorrison opened this issue 2 years ago · comments

Description

Create a new fluent-assertions library within aws-cdk. This offers fluent chained filtering of resources (and other fields) on a template, allowing a test author to more clearly represent what they want to test.

Roles

Role	User
Proposed by	@sgmorrison
Author(s)	@sgmorrison
API Bar Raiser	@MrArnoldPalmer
Stakeholders	@MrArnoldPalmer, @rix0rrr

See RFC Process for details

Workflow

Author is responsible to progress the RFC according to this checklist, and
apply the relevant labels to this issue so that the RFC table in README gets
updated.

Steven Morrison · Answer 1 · Thu Oct 06 2022 04:09:24 GMT+0800 (China Standard Time)

Draft of RFC

The intent is to add new methods to template.ts in assertions module. These methods all for functional-style chained filtering of resources (and other fields on a template), allowing a test author to more clearly represent what they want to test.

Working Backwards

The proposed change would result in additional content in the existing assertions/README.md:

Resource Matching & Retrieval

Beyond resource counting, the module also allows asserting that a resource with
specific properties are present.

The following code asserts that the Properties section of a resource of type
Foo::Bar contains the specified properties -

template.hasResourceProperties('Foo::Bar', {
  Foo: 'Bar',
  Baz: 5,
  Qux: [ 'Waldo', 'Fred' ],
});

Alternatively, if you would like to assert the entire resource definition, you
can use the hasResource() API.

template.hasResource('Foo::Bar', {
  Properties: { Foo: 'Bar' },
  DependsOn: [ 'Waldo', 'Fred' ],
});

-- New content begins --

APIs also exist that allow functional exploration of a template, with filterByType, filter, and so on. Using the functional APIs lets you separate code for template filtering from the assertions you make about the template content.

// Restrict to only the alarms we care about
const barAlarms = template
    .resources()
    .filterByType('AWS::Alarm')
    .filter({
        Properties: { Foo: 'Bar' },
    });

expect( barAlarms.size() ).toBe( 4 );

// Property we want to be true on all extracted resources
const foopAlarms = barAlarms.filter({
    Properties: { Baz: 'Foop' },
})

expect( barAlarms ).toBe( foopAlarms );

These APIs also allow expression of tests not possible with the simpler hasResourceProperties, such as demonstrating that a subset of resources all match a given criteria (for example, that all Beta alarms notify to SNS while all Prod alarms notify by email).

// Restrict to only the alarms we care about
const barAlarms = template
    .resources()
    .filterByType('AWS::Alarm')
    .filter({
        Properties: { Foo: 'Bar' },
    });

expect( barAlarms.size() ).toBe( 4 );

// Property we want to be false on all extracted resources
const foopAlarms = barAlarms.filter({
    Properties: { Baz: 'Foop' },
})

expect( foopAlarms ).toBeEmpty();

-- New content ends --

Beyond assertions, the module provides APIs to retrieve matching resources.
The findResources() API is complementary to the hasResource() API, except,
instead of asserting its presence, it returns the set of matching resources.

Public FAQ

What are we launching today?

A new set of APIs on the existing Assertions module.

Why should I use this feature?

This change makes writing readable unit tests easier, and gives a paradigm that developers are familiar with for interacting with their template in tests.

Internal FAQ

Why are we doing this?

As a developer, I frequently find that unit tests I write for CDK stacks are hard to craft and make readable. The statement of what I want to test is often mixed in with setting up the world I expect to exist, and want the test to take as a given. I also find it difficult to assert the negative of some things. For example, it is easy to say a resource exists with given properties, but harder to say a resource does not exist with given properties.

Why should we not do this?

The same functionality is not available today. The downside to implementing this is that the Template class will have two sets of methods for interacting with resources and other facets. This may confuse users.

Is this a breaking change?

No

Rico Hermans · Answer 2 · Tue Oct 11 2022 16:02:31 GMT+0800 (China Standard Time)

It seems your proposed API does more or less the same as what findResources does, except with a fluent API style instead of all in one method call.

Is that right?

Steven Morrison · Answer 3 · Fri Oct 14 2022 17:56:16 GMT+0800 (China Standard Time)

It seems your proposed API does more or less the same as what findResources does, except with a fluent API style instead of all in one method call.

Is that right?

The fluent API gives more capabilities than findResources. If you look at both examples I present, I use the fluent API to find a subset of resources, make an assertion on them, then perform further filtering on that subset. The output of findResources is a Map of string to any. It is possible to continue working with the Map to filter and assert, but the API you are working with now is very different to what Template offers, and the code becomes less readable. I've tried to re-implement my example below with current APIs to show the difference.

// Restrict to only the alarms we care about
const barAlarms:  { [key: string]: { [key: string]: any } } = template.findResources('AWS::Alarm', {
    Properties: { Foo: 'Bar' },
});

expect( Object.keys(barAlarms).length ).toBe( 4 );

// Property we want to be false on all extracted resources
for (const barAlarmKey in barAlarms) {
    for (const barAlarm in barAlarms[barAlarmKey]) {
        if (barAlarm === "Properties") {
            for (const elementKey in barAlarms[barAlarmKey][barAlarm]) {
                if (elementKey === "Baz") {
                    assert(barAlarms[barAlarmKey][barAlarm][elementKey] !== "Foop");
                }
            }
        }
    }
}

Steven Morrison · Answer 4 · Mon Oct 17 2022 14:39:31 GMT+0800 (China Standard Time)

@rix0rrr , @MrArnoldPalmer - would either of you be willing to BR this RFC?

Mitchell Valine · Answer 5 · Tue Oct 18 2022 03:59:38 GMT+0800 (China Standard Time)

I talked briefly with @rix0rrr about this earlier and here are some things we talked about.

Chainable/fluent style APIs can sometimes be difficult to implement while staying JSII compliant. @sgmorrison have you been designing around those requirements at all and have you run into any problems if so? I don't believe anything about the examples presented is indicative of JSII targets not being supported but we should be aware of what these APIs will look like across all languages.
Can the subset example not be achieved like so:

template.resourcePropertiesCountIs('AWS::Alarm', {
  'Foo': 'Bar',
  'Baz': Match.not('Foop'),
}, 4);

I definitely can see the usefulness of making resource filtering multi-step, so you can filter your results multiple times with less code repetition. However, since we haven't seen a lot of requests to add capabilities/apis to the assertions library, and adding these capabilities likely would require some number of development -> testing -> feedback -> iteration cycles, I think developing this in a separate package would be ideal. This also would let us gauge interest from the community/users for inclusion within @aws-cdk/assertions.

Would you be willing to create this library and publish it (to all JSII targets) so we can look at it more closely, make suggestions etc, while allowing cdk users to give it a shot as well? If so, documentation should automatically be published to constructs.dev and projen can be used to automate the setup of the repository and publishing actions, which should minimize the amount of procedural stuff you need to do get it going.

Steven Morrison · Answer 6 · Wed Oct 19 2022 15:28:58 GMT+0800 (China Standard Time)

Chainable/fluent style APIs can sometimes be difficult to implement while staying JSII compliant. @sgmorrison have you been designing around those requirements at all and have you run into any problems if so?

Good call-out. I've only prototyped the APIs in TypeScript so far, and have had issues getting my development environment setup to confirm that changes work with JSII. I've also focused energy on the RFC and BR pieces. I'll keep this in mind as implementation progresses.

Can the subset example not be achieved like so...

Yes. The issue the example is highlighting is that if we assert the subset condition with a separate properties block like this, we run the risk of mismatches between the first and second calls to template. We also are forced to specify all of our conditions together, which can make the test harder to read. If we want our test to say "there are exactly 4 alarms, and none have Bar:Foop", we are forced to instead write it as "there are 4 non-Bar:Foop alarms". The intent is less clear, making the test harder to understand.

I think developing this in a separate package would be ideal.

Got it. I expect some of the implementation will be duplicative as I won't have access to the internal TemplateType field on Template, but we can work around this in the short term.

Would the separate package still be within the aws-cdk project, or initially stand-alone?

Before implementing, I'll refresh the RFC based on the new approach (working backwards from a new fluent-assertions package instead of adding to assertions). Are you willing to BR this, @MrArnoldPalmer ?

Mitchell Valine · Answer 7 · Wed Oct 19 2022 22:50:23 GMT+0800 (China Standard Time)

The intent is less clear, making the test harder to understand.

Definitely agree with this, though of course to reduce duplication users could do some object ...spread stuff if needed I supposed though that's still not as explicit, IE:

const hasTheseProps = { Foor: 'Bar' };
template.resourcePropertiesCountIs('AWS::Alarm', hasTheseProps, 4);
template.resourcePropertiesCountIs('AWS::Alarm', {
  ...hasTheseProps,
  'Baz': 'Foop',
}, 0);

Anyway, for now I think this should be vended from a separate repository and published under whatever name you think is best from your npm/maven/pypi/github account, mostly to avoid us blocking you for reviews/feedback during early stage development and iteration. There may be some small duplication (which is fine for now) but you may be able to get pretty far taking a dependency on and extending the public interfaces of @aws-cdk/assertions. Additionally if exporting additional types from @aws-cdk/assertions makes writing your own assertion utilities easier I'm sure we would be in favor of doing that. No reason to block users from extending the functionality we have as long as we know those APIs will be relatively stable.

I'm happy to BR this proposal but since this will be vended separately I'd say don't worry too much about waiting for our review on anything, but do of course reach out for help/guidance and I will provide what I can. I'm more curious to see the result of what you think is a superior UX/API (with JSII support ideally) than making it fit the general design/standards of the current testing tools in the aws/aws-cdk repo. Even if the result doesn't end up fitting into the core repo/packages, high quality alternatives with different design philosophies are super useful, arguably more so.

Steven Morrison · Answer 8 · Fri Oct 21 2022 13:28:31 GMT+0800 (China Standard Time)

Got it, thanks for the help.

Momo Kornher · Answer 9 · Sat Nov 04 2023 01:33:16 GMT+0800 (China Standard Time)

Marking this RFCs as rejeceted as per this comment. We appreciate the effort that has gone into this proposal. Marking an RFCs as rejected is not a one-way door. If the circumstances have changed, and/or a substantial change was made to the proposal, please open a new issue/RFC. You might also consider raising a PR to aws/aws-cdk directly or self-publishing to Construct Hub.