Adding scout APM turns bad code into an error on production only
timdiggins opened this issue · comments
Bad experience with Scout APM:
I added Scout APM (gem) to an existing longstanding but well-tested codebase, pushed to CI, all green. Deployed to production, and started then got some quite unexpected errors on code that had never errored before.
It turns out these were (mistakenly) breaking a Rails API contract, but the difference was the insertion of Scout APM into the code stack.
Here's a simple enough repro:
timdiggins/scout-monitor-raises-error@0205cc8
Now on the one hand - great news that scout has helped find some errors in the legacy code (which would prevent gem / ruby upgrades to be sure).
But the big problem here is that the instrumentation doesn't happen in the the test environment by default. Don't get me wrong - I don't want to report to scout from the test environment, but we want the call stack to be as similar as possible in production and test.
Am I missing something here? It seems like the best bet is to have SCOUT_MONITOR=true in test (especially CI) and this won't actually report to the scout backend anyway.
Hi @timdiggins You're correct that you'll need to set SCOUT_MONITOR=true in any environment, including test environment, if you want the instruments to be applied.
Leave the SCOUT_KEY empty, or set SCOUT_HOST=http://localhost (or both) to prevent it from showing up in your dashboard on Scout's UI.
@dlanderson thanks for clarification, but I'm not sure this is ready for closure.
I think the docs (both here and on scout website) should be changed to have monitor: true in test (and development) to ensure production parity and prevent unexpected errors in production only?
What would be the argument for not instrumenting test/development but only instrumenting production?