DevOps example not working correctly
FrankHoogmans opened this issue · comments
Which Renovate are you using?
Renovate Open Source CLI
Which platform are you using?
Azure DevOps
Have you checked the logs? Don't forget to include them if relevant
Every single repo gives the following error (redacted repo names):
"renovateVersion": "23.74.1"
ERROR: Repository has unknown error (repository=*******)
"err": {
"statusCode": 401,
"result": {
"$id": "1",
"innerException": null,
"message": "TF400813: The user '' is not authorized to access this resource.",
"typeName": "Microsoft.TeamFoundation.Framework.Server.UnauthorizedRequestException, Microsoft.TeamFoundation.Framework.Server",
"typeKey": "UnauthorizedRequestException",
"errorCode": 0,
"eventId": 3000
},
"message": "TF400813: The user '' is not authorized to access this resource.",
"stack": "Error: TF400813: The user '' is not authorized to access this resource.\n at RestClient.<anonymous> (/home/vsts/.npm/_npx/2374/lib/node_modules/renovate/node_modules/typed-rest-client/RestClient.js:202:31)\n at Generator.next (<anonymous>)\n at fulfilled (/home/vsts/.npm/_npx/2374/lib/node_modules/renovate/node_modules/typed-rest-client/RestClient.js:6:58)\n at processTicksAndRejections (internal/process/task_queues.js:93:5)"
}
INFO: Repository finished (repository=*******)
"durationMs": 402
What would you like to do?
We are using Renovate in an Azure DevOps pipeline, based on the starter template available at https://docs.renovatebot.com/setup-azure-devops/. This runs multiple times a day, to update dependencies for all of our modules (repositories).
Last friday (6th of november) everything worked great, but yesterday I started getting 401's on the DevOps repository access. I have not changed the configuration in any way, and even tried the template in another devops account to be sure it wasn't account related. Downgrading to the version that last worked did also not provide a working solution (same errors).
I have checked, and do not have any limitations on the System.AccessToken about using it in scripts or across repositories.
I am unsure if this is somehow a bug in renovate, or if i'm doing anything wrong with the configuration, but given that the example doesn't work for me anymore, gives me reason to think something changed, but I seem to be unable to figure out what.
@JamieMagee do you think this could be related to the transitive dependency updates yesterday?
I also recall previously problems of expired tokens but this error message doesn't warn about that.
I also recall previously problems of expired tokens but this error message doesn't warn about that.
I have already tried swapping tokens (switching to a PAT for my own account instead of System.AccessToken). Creating a new pipeline with the same yaml, and trying the template on a different devops account. So expired tokens don't sound likely.
@JamieMagee do you think this could be related to the transitive dependency updates yesterday?
I don't think so. The lockfile updates went out yesterday, but this issue started happening on Friday.
I am also seeing this on my own Renovate jobs running against Azure DevOps (I should check in on those more often!). They started failing on Friday between 19:00 and 20:00 CET, but the Renovate version was 23.73.2 for both. It's still failing right now.
This might be something on the Azure DevOps side. Let me investigate it.
@JamieMagee do you think this could be related to the transitive dependency updates yesterday?
I don't think so. The lockfile updates went out yesterday, but this issue started happening on Friday.
I am also seeing this on my own Renovate jobs running against Azure DevOps (I should check in on those more often!). They started failing on Friday between 19:00 and 20:00 CET, but the Renovate version was
23.73.2for both. It's still failing right now.This might be something on the Azure DevOps side. Let me investigate it.
I also did some more digging, but have not found anything so far. I'll just share what I checked.
The Azure DevOps release notes seem to indicate nothing has changed on Friday the 6th of November:
https://docs.microsoft.com/en-us/azure/devops/release-notes/2020/sprint-177-update
Last release of anything seems to have been October 21st.
I checked the run logs between a run that worked on friday, and one that failed on monday.
Both runs are using exactly the same software on the build agent (ubuntu image, devops agent software, bash and npmauthenticate task versions). This is the 'Initialize job' logging:
Agent name: 'Hosted Agent'
Agent machine name: 'fv-az101-655'
Current agent version: '2.175.2'
Operating System
Ubuntu
18.04.5
LTS
Virtual Environment
Environment: ubuntu-18.04
Version: 20201102.0
Included Software: https://github.com/actions/virtual-environments/blob/ubuntu18/20201102.0/images/linux/Ubuntu1804-README.md
Current image version: '20201102.0'
Agent running as: 'vsts'
Prepare build directory.
Set build variables.
Download all required tasks.
Downloading task: Bash (3.177.2)
Downloading task: npmAuthenticate (0.174.0)
Checking job knob settings.
Knob: AgentToolsDirectory = /opt/hostedtoolcache Source: ${AGENT_TOOLSDIRECTORY}
Knob: AgentPerflog = /home/vsts/perflog Source: ${VSTS_AGENT_PERFLOG}
Finished checking job knob settings.
Start tracking orphan processes.
So same version of Renovate and (apparent) same version of Azure DevOps, but it suddenly stopped working?
Can somebody log at debug level in case it gives additional info? I'm guessing it's failing at one of these lines: https://github.com/renovatebot/renovate/blob/ad50398fb3981a7c5d04b96bddce3dbad45a7d21/lib/platform/azure/index.ts#L124-L125
This question was coincidentally asked yesterday: https://stackoverflow.com/questions/64759524/tf400813-the-user-is-not-authorized-to-access-this-resource-when-using-syste
Someone asked the same question 4 months ago, here was some suggestion from a MSFT employee:
Note that the OP said that the problem resolved itself though.
So same version of Renovate and (apparent) same version of Azure DevOps, but it suddenly stopped working?
Correct. Although at first the renovate version was different, but reverting doesn't fix it.
This question was coincidentally asked yesterday: https://stackoverflow.com/questions/64759524/tf400813-the-user-is-not-authorized-to-access-this-resource-when-using-syste
Someone asked the same question 4 months ago, here was some suggestion from a MSFT employee:
Note that the OP said that the problem resolved itself though.
I agree it seems to be a DevOps (authentication) issue... We use the Microsoft provided build agents and images, no proxy settings, and (I guess) 0 debugging on network issues there for us. I can't seem to find any other reports regarding these issues...
I still hope it will resolve itself automagically...
Can somebody log at debug level in case it gives additional info? I'm guessing it's failing at one of these lines: https://github.com/renovatebot/renovate/blob/ad50398fb3981a7c5d04b96bddce3dbad45a7d21/lib/platform/azure/index.ts#L124-L125
Ok, a debug run seems to confirm that it fails somewhere after 'initRepo'.
This is the debug-logging of a single repo (redacted repo name)
2020-11-11T10:12:20.2543975Z DEBUG: Using default gitAuthor: Renovate Bot <renovate@whitesourcesoftware.com>
2020-11-11T10:12:20.2544209Z DEBUG: Using baseDir: /tmp/renovate
2020-11-11T10:12:20.2544385Z DEBUG: Using cacheDir: /tmp/renovate/cache
2020-11-11T10:12:20.2544787Z DEBUG: Initializing Renovate internal cache into /tmp/renovate/cache/renovate/renovate-cache-v1
2020-11-11T10:12:20.2545019Z DEBUG: Commits limit = null
2020-11-11T10:12:20.2545204Z INFO: Repository started (repository=***)
2020-11-11T10:12:20.2545405Z "renovateVersion": "23.77.0"
2020-11-11T10:12:20.2545663Z DEBUG: Using localDir: /tmp/renovate/repos/azure/*** (repository=***)
2020-11-11T10:12:20.2545983Z DEBUG: initRepo("***") (repository=***)
2020-11-11T10:12:20.6909588Z ERROR: Repository has unknown error (repository=***)
2020-11-11T10:12:20.6909994Z "err": {
2020-11-11T10:12:20.6910525Z "statusCode": 401,
2020-11-11T10:12:20.6910718Z "result": {
2020-11-11T10:12:20.6910898Z "$id": "1",
2020-11-11T10:12:20.6911119Z "innerException": null,
2020-11-11T10:12:20.6912043Z "message": "TF400813: The user '' is not authorized to access this resource.",
2020-11-11T10:12:20.6912504Z "typeName": "Microsoft.TeamFoundation.Framework.Server.UnauthorizedRequestException, Microsoft.TeamFoundation.Framework.Server",
2020-11-11T10:12:20.6912984Z "typeKey": "UnauthorizedRequestException",
2020-11-11T10:12:20.6913193Z "errorCode": 0,
2020-11-11T10:12:20.6913365Z "eventId": 3000
2020-11-11T10:12:20.6913509Z },
2020-11-11T10:12:20.6913925Z "message": "TF400813: The user '' is not authorized to access this resource.",
2020-11-11T10:12:20.6915288Z "stack": "Error: TF400813: The user '' is not authorized to access this resource.\n at RestClient.<anonymous> (/home/vsts/.npm/_npx/2588/lib/node_modules/renovate/node_modules/typed-rest-client/RestClient.js:202:31)\n at Generator.next (<anonymous>)\n at fulfilled (/home/vsts/.npm/_npx/2588/lib/node_modules/renovate/node_modules/typed-rest-client/RestClient.js:6:58)\n at processTicksAndRejections (internal/process/task_queues.js:93:5)"
2020-11-11T10:12:20.6916019Z }
2020-11-11T10:12:20.6919136Z DEBUG: Unknown res (repository=***)
2020-11-11T10:12:20.6919594Z "res": "unknown-error"
2020-11-11T10:12:20.6931906Z DEBUG: Repository timing splits (milliseconds) (repository=***)
2020-11-11T10:12:20.6932204Z "splits": {},
2020-11-11T10:12:20.6932363Z "total": 463
2020-11-11T10:12:20.6936525Z INFO: Repository finished (repository=***)
2020-11-11T10:12:20.6936775Z "durationMs": 463
I figured out the issue. It's a little bit convoluted but bear with me.
On Friday 6th November, version 1.8.0 of typed-rest-client was published. It included this commit about disabling authentication on redirects.
Looking at the changes on that commit, the authentication is not added to every request, so we're authenticating as an anonymous user for some requests (where have I seen that error 🤔)
azure-devops-node-api has a dependency on typed-rest-client@^1.7.3. So, even though Renovate specifies a different version in our yarn.lock file, because our guide use npx the yarn.lock isn't used. The ^1.7.3 range resolves to 1.8.0, which is currently broken.
Therefore, the fix for this should either be in azure-devops-node-api to set allowCrossOriginAuthentication, or in typed-rest-client to add authentication to all requests.
I continued the discussion here
@JamieMagee do we need to make any code modifications now?
Yes, small code change required.
I can send a PR out later this evening, or tomorrow. I'm on mobile right now.
I can confirm version 23.83.3 of renovatebot does work as expected again. Thank you so much for the help 👍