Proposal: use native Go test framework only in functional tests
artursouza opened this issue · comments
Our functional test framework in written on top of the existing Go's test framework but it has its own custom interface. It makes local development and debugging harder because we lose IDE integration that we get with unit and E2E tests (VS Code, for example). Another disadvantage is that the functional tests compile the binaries in a separate logic than we have in our Makefiles, so we need to keep the compilation logic in two places when there is a change.
Advantages of this proposal:
- Native IDE integration for running functional tests locally (a must IMO)
- Consistency in building Dapr binaries (nice to have IMO)
- No new framework to learn to write, run and debug functional tests (nice to have IMO)
I cannot think of any disadvantage apart from the rework, maybe having to compile the binaries before running functional tests, which can be seen as a disadvantage depending on the point of view, but since we are running binaries, it is fair game IMO. For example, users need to compile Docker images to run E2E tests.
Again, the main motivation for this proposal is item (1) and I can compromise on (2) and (3).
Current code for functional tests (no "run" or "debug" shortcut in VS Code):
Current code for E2E tests (you can run or debug a single test straight from VS Code):
There is one way to debug from VS Code, it involved putting a break-point at the test you want to debug and run all tests from here, eventually it hits your code.
Native IDE integration for running functional tests locally gets a +1 from me. It's easier for developers to be able to use breakpoints naturally rather than put panics in places and output the vals that way. Both work, but in concurrent systems its nice to use the debugger. This also leans to being more first time contributor friendly. I know several draft PRs that have been a work in progress on error standardization efforts, but once at the IT part they hit a road block. This might help there too.
Integration tests are designed so that we never have a situation where code is changed, but these changes are not reflected in the compiled code that is tested against resulting in false positives or false negatives. This is nuclear imo, and is an issue today with e2e tests. It is intentional that running tests need only be done with a single command. IDEs can be configured to run custom flags when testing, so a custom -focus flag can be given to choose your area of interested tests.
E2E tests are not dev friendly today, and find myself heavily modifying the framework in order to run them with time.Sleeps and kubectl port-forwards to actuate.
@cicoyle can you expand a bit more on how'd you want to use a debugger rather than intentionally adding panics in the code- do IDEs today not allow for adding breakpoints to compiled binaries? Tbh, I don't know. It would be true for e2e in any case.
The issue I see currently for contributors writing integration tests are that in-memory/file based components are not available for some APIs, rather than a hickup on writing int tests themselves.