grosser / kennel

Datadog monitors/dashboards/slos as code, avoid chaotic management via UI

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

"Duplicate of an existing monitor_id:..." should cause the syncer to restart

zdrve opened this issue · comments

The above error seems to be the result of a race condition of some kind (I can't say with complete confidence yet what's happening). Very often, it seems that the monitor ID it reports is effectively exactly the same one that it's trying to create (i.e. same kennel tracking id).

If we see the "duplicate of an existing monitor" error, then we should restart the syncer update from the top – but only do this once, to avoid looping.

never saw that being part of a race condition 😞
no idea what the root cause could be ... except for someone reusing constants for kennel ids and then changing them