`newrelic deployments` does not work with newly created API keys
Arkham opened this issue · comments
newrelic deployments only work with the original license key that came with the account.
Description
We have recently cycled our NewRelic License Key and updated config/newrelic.yml.
Running newrelic deployments -e production -r 1234567890 'Testing new api key' now yields:
Deployment not recorded: Forbidden
After reverting the key inside config/newrelic.yml to the previous INGEST - LICENSE key the command works again.
Expected Behavior
Should record the deployment.
Troubleshooting or NR Diag results
Posting to the API manually works with the newly created key:
curl -X POST "https://api.newrelic.com/v2/applications/${MY_APPI_ID}/deployments.json" \
-H "Api-Key: ${NEW_RELIC_ACCESS_KEY}" \
-i \
-H "Content-Type: application/json" \
-d \
'{
"deployment": {
"revision": "1234567890",
"changelog": "",
"description": "Testing new api key",
"user": "Arkham",
"timestamp": "2021-07-07T17:26:00Z"
}
}'
Hello @Arkham,
I tried using the newrelic deployments command with my own test app, and I also saw Deployment not recorded: Forbidden with a valid license key. For me it doesn't appear to be working correctly at all, with any of the license keys I currently use. I also tried using that curl command with my test app info, but that also didn't work for me, I received back {"error":{"title":"No API key specified"}}. So maybe I'm just doing something wrong there?
I'm not very familiar with our rest API, so I'll need to reach out to that team and see if they can help me understand what's going on here and why our rake command isn't working properly.
Thank you for bringing this to our attention
I've spoken with the team that handles our API backend and learned some more about what's going on.
So basically, the API that the rake task is using is actually going to a different endpoint than the manual API does. There are also a couple types of license keys, and it looks like the API that the rake command communicates with wasn't updated on the backend to accept one of these newer types of license keys, which is why the rake task fails with certain license keys. They weren't aware of the issue either, so thank you very much for reporting this problem, they are looking into the changes needed to solve the issue.
In the meantime, if anyone else encounters this issue, the best workaround for now is to use the API manually to record your deployments until the API backend is updated. Please see the API documentation here for more details about how to use this API.
(Internal Ticket - link)
We are keeping this open for now to provide awareness of the issue. This issue will be closed when the internal ticket closes.
Hi! Is there any progress on this issue?
Hi @sy-hash. Thanks for following up on this one. Here are some updates:
- The "Posting to the API manually" with
curlworkaround discussed above in this GitHub Issue works because it targets an entirely different endpoint. - The
newrelic deploymentscommand targets our "rpm" subdomain which was originally designed to work with one type of license key and does not work with an API type key. This is ultimately why @Arkham was unable to use the new key with the command. - Conversely, the manual
curlcommand mentioned targets the "api" subdomain. That subdomain will not work with the other type of license keys and requires API type keys. This is ultimately why @tannalynn was unable to getcurlto work earlier. - Beyond the use of different subdomains, the
newrelic deploymentscommand and thecurlcommand differ in that the first one uses an XML endpoint and the other uses a JSON endpoint. On the backend these have implementation details that differ a bit beyond just the request/response payload formatting. - The team responsible for the "rpm" subdomain based endpoints has looked into what it take to support the use of the newer API style keys within an XML payload and has discovered a considerable amount of effort involved. Work on this front is ticketed internally and awaiting prioritization. I do not have an ETA for this work.
- The Ruby agent team responsible for maintaining the
newrelic deploymentscommand can modify the command to leverage the "api" subdomain and JSON based endpoint to effectively do the same thing thecurlcommand is able to do. The tricky part there is that we would need to conditionally or dynamically come up with a way to support both types of license keys, because both backend endpoints require a specific type and reject the other type. - In addition to the
newrelic deploymentscommand included with the Ruby agent and thecurlcommand mentioned in this GitHub Issue, there is an additional way to report deployments available. The newrelic-cli tool supports anewrelic apm deployment createcommand (usage info here).
I will discuss these updates with the rest of the Ruby agent team and if we have anything additional to add we will share it here. Thank you very much for your interest and patience.
@fallwith
Thanks for the detailed information!
I will try the workaround you proposed.
Maintainers: The Python agent makes use of an api_key parameter in newrelic.ini, so we could easily add that to our newrelic.yml.
I'm thinking we have it work like this:
api_keyis present innewrelic.yml: point the net call to the "api" subdomainapi_keyis missing innewrelic.yml: continue to point to the "rpm" subdomain- anticipate "Forbidden" errors for the "rpm" subdomain and mention in the error output that the API key approach is available
