design: tech debt: e2e tests refactoring

Question

tzaffi opened this issue a year ago · comments

Problem

The end to end tests are an important sanity check ensuring that the latest version of indexer is production-ready.

However, they suffer from a few issues that make them difficult to work with and identify/fix problems when they occur:

weave together disparate components:
- AWS tarballs to define a network to test against
- docker containers for test environmenting
- python library to execute processes and validate results

These components have become bloated:

docker:
- over-duplication across multiple docker-compose.yml's and Dockerfile's
python:
- old and bespoke tests that are very difficult to modify and modernize
- favor speed (via multi-processing) over clarity and ease of use
- xrun may not even work correctly. @bbroder-algo pointed out in a discussion in #1523 that the time out may not engage: https://stackoverflow.com/questions/36952245/subprocess-timeout-failure

As a result:

adding new tests to precisely validate new features becomes tedious and basically isn't done
diagnosing and fixing a break can take hours/days instead of minutes for the uninitiated

Produce a design doc that proposes a way to refactor the end to end tests so as to fix these issues.

None

Medium

Will Winder · Answer 1 · Sat May 20 2023 04:24:47 GMT+0800 (China Standard Time)

Maybe these old tests can be deprecated in favor of the new e2e scripts used for conduit: https://github.com/algorand/conduit/tree/master/e2e_tests

Zeph Grunschlag · Answer 2 · Sat May 20 2023 04:29:09 GMT+0800 (China Standard Time)

Maybe these old tests can be deprecated in favor of the new e2e scripts used for conduit: https://github.com/algorand/conduit/tree/master/e2e_tests

They seem to share some python functions, but I agree that we should look to see if we can consolidate the python if possible.