Test Pyramid doubts

[Michał Pietrus]

Hi @searls ,

some time ago I've watched yours Game of Life series and finally landed here – thanks for the effort! I'm just curious about the intention of Test Pyramid comments, especially the one concerning integration tests (or something between E2E and units). In particular, where would you classify testing collaborations? Or perhaps the meaning is: do not couple the test with particular framework or more general, a piece of code that I don't own?

[Justin Searls]

Thanks for asking this @blelump (excuse me immediately closing it for the sake of tidiness).

I'll be honest: I did not think through this criticism in the same light you did. In fact, the first time this issue really came to mind was when I was doing a workshop at the SCNA conference in Los Angeles last month. That's my own fault for using the "agile test pyramid" as a whipping boy for so many years without critically thinking about it.

What you said is basically right. How I would draw what I actually do now:

          /\
         /  \  <-- integrated tests for confidence the system works
        /____\
       /      \  <-- (a) collaboration tests for design benefit + maybe (b) narrowly-defined semi-integrated tests that handle a class of units very consistently
      /________\
     /          \  <--- isolated unit tests of (hopefully) pure functions for confidence they work
    /____________\

So I think you had it right in your speculation above. Thanks for asking! I'll be more careful of this in the future.

[Julian Berman]

(For my own clarity) Is the distinction being made to just not be so dogmatic about the middle and accepting that sometimes there are collections of units that seem solid enough to safely isolate the collaboration and not worry that you'll regret it later, or am I totally missing the point?

[Justin Searls]

Correct, there are two types of tests that are fine in the middle:

1. Collaboration tests that isolate themselves from the layers beneath, offering a design benefit and no risk of costly redundant code coverage
2. Targeted test suites against a common class of units (e.g. controller objects) that would be much more efficient to test in a careful and consistent manner than piling on more integrated tests. These have to be carefully defined and vetted for consistency, or else you risk the toxic hellstew of ambiguity in what gets tested, what's left real vs. faked, and so on.