mozilla / experimenter

A web application for managing user experiments for Mozilla Firefox.

Home Page:https://experimenter.services.mozilla.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Refactor Jetstream client tests

mikewilli opened this issue · comments

There is a lot of static test data that turns any slight schema change into (at best) a huge multi-thousand-line PR, or (at worst) a nightmare of unclear test data updates. We should try to refactor this situation into something more manageable.

I think that the ideal outcome of this would be to have well-defined schemas in Pydantic, and use the library that the schemas package already has to generate random test data as a sort of fuzzing test. Then we can also have a folder of known good and bad JSON files of the inputs and make sure they pass or fail during ingestion, and produce the expected output. When there are new failures in production we can add those to the known good folder and update ingestion to work with them, and then the tests will pick this up automatically. This is basically how nimbus-shared and now nimbus-schemas works for testing other shared schemas.

┆Issue is synchronized with this Jira Task