Tracking unit test coverage

[Marilia Melo]

TL;DR

Tracking issue to increase unit tests for validation failures, permissions, failed sms messages w/ rollback, etc

Expected behavior
Most critical use cases of our system should be covered by unit tests.

Observed behavior
e2e tests cover some, but not all use cases.

Additional information
Filing this issue for tracking this work that may be deprioritized in name of new features.

[Seth Vargo]

I actually think this is done as of #1372 (#1296). Let me know if you disagree.

[Weston Haught]

The db coverage is great, but I think we can go further to cover the controller layer better. Assert things like HTTP error codes / error enums are correct. Eg. several IssueCode scenarios or the recent 500s on redirector (which I think you also added coverage for)

[Seth Vargo]

Controller tests aren't unit tests though - most are gonna be clicky-clicky. I'm also generally against issues that have no clear success criteria. If we want to open specific issues for specific packages, that will ensure we have the coverage and people can parallelize working on them.

[Mike Helmick]

We could set a target for test coverage for the whole project and attempt to meet it. Not sure if clickyclicky gets go coverage stats.

setting an initial benchmark of 70%?

[Seth Vargo]

internal/i18n	0.033s	coverage: 0.0% of statements
internal/project	0.029s	coverage: 30.8% of statements
internal/routes	0.187s	coverage: 20.4% of statements
pkg/api	0.034s	coverage: 41.2% of statements
pkg/cache	2.555s	coverage: 62.6% of statements
pkg/controller/admin	24.209s	coverage: 51.4% of statements
pkg/controller/apikey	23.223s	coverage: 23.4% of statements
pkg/controller/appsync	16.547s	coverage: 60.6% of statements
pkg/controller/associated	29.159s	coverage: 85.2% of statements
pkg/controller/codes	22.016s	coverage: 12.6% of statements
pkg/controller/issueapi	0.135s	coverage: 8.4% of statements
pkg/controller/mobileapps	22.923s	coverage: 24.1% of statements
pkg/controller/modeler	15.662s	coverage: 72.0% of statements
pkg/controller/realmadmin	23.122s	coverage: 29.5% of statements
pkg/controller/redirect	17.269s	coverage: 87.4% of statements
pkg/controller/user	28.669s	coverage: 21.8% of statements
pkg/database	31.768s	coverage: 53.4% of statements
pkg/otp	24.010s	coverage: 72.0% of statements
pkg/rbac	0.024s	coverage: 9.5% of statements

[Weston Haught]

hmm those stats... could use improvement

[Seth Vargo]

Update

internal/i18n              coverage: 0.0% of statements
internal/project           coverage: 16.7% of statements
internal/routes            coverage: 21.0% of statements
pkg/api                    coverage: 36.8% of statements
pkg/cache                  coverage: 62.8% of statements
pkg/controller/admin       coverage: 51.4% of statements
pkg/controller/apikey      coverage: 93.7% of statements
pkg/controller/appsync     coverage: 60.8% of statements
pkg/controller/associated  coverage: 85.2% of statements
pkg/controller/codes       coverage: 12.6% of statements
pkg/controller/issueapi    coverage: 85.7% of statements
pkg/controller/login       coverage: 5.8% of statements
pkg/controller/middleware  coverage: 61.2% of statements
pkg/controller/mobileapps  coverage: 93.0% of statements
pkg/controller/modeler     coverage: 72.0% of statements
pkg/controller/realmadmin  coverage: 84.3% of statements
pkg/controller/redirect    coverage: 89.4% of statements
pkg/controller/user        coverage: 52.4% of statements
pkg/database               coverage: 59.2% of statements
pkg/rbac                   coverage: 61.1% of statements

[Seth Vargo]

internal/i18n              92.5% of statements
internal/project           88.6% of statements
internal/routes            94.6% of statements
pkg/api                    91.9% of statements
pkg/cache                  82.7% of statements
pkg/controller/admin       56.1% of statements
pkg/controller/apikey      93.8% of statements
pkg/controller/appsync     53.6% of statements
pkg/controller/associated  85.2% of statements
pkg/controller/cleanup     70.9% of statements
pkg/controller/codes       16.6% of statements
pkg/controller/issueapi    79.1% of statements
pkg/controller/login        5.8% of statements
pkg/controller/middleware  60.6% of statements
pkg/controller/mobileapps  93.0% of statements
pkg/controller/modeler     72.0% of statements
pkg/controller/realmadmin  81.5% of statements
pkg/controller/realmkeys   36.7% of statements
pkg/controller/redirect    85.7% of statements
pkg/controller/rotation    61.1% of statements
pkg/controller/smskeys     78.5% of statements
pkg/controller/user        52.5% of statements
pkg/database               63.3% of statements
pkg/rbac                   61.1% of statements
pkg/signatures             91.7% of statements

Looks like we need to focus on the codes and login packages @whaught @mikehelmick

[Weston Haught]

pkg/controller/codes	65.4% of statements
pkg/controller/login	65.9% of statements

[Seth Vargo]

Nice! We should take a stab at:

• appsync
• rotation
• database
• user
• rbac

to see if there's any obviously low-hanging fruit. If not, I think we can close this out then.

[Seth Vargo]

I feel comfortable where we've landed on this. Let's shift focus to the key server.