Renovate not working for SBT private repositories
marius-carp opened this issue · comments
What Renovate type, platform and version are you using?
bitbucket, self-hosted
Describe the bug
Private repositories don't work with SBT. Looking into renovate logs I don't see it looking for the private repositories. I just see logs of public repositories.
Relevant debug logs
Dependency extraction complete (repository=sef/renovate-test-scala)
"baseBranch": "master",
"stats": {
"managers": {"sbt": {"fileCount": 1, "depCount": 1}},
"total": {"fileCount": 1, "depCount": 1}
}
Config
{
"endpoint": "https://bitbucket.domain.tools",
"platform": "bitbucket-server",
"username": "username",
"password": "password",
"extends": [
"config:base"
],
"repositories": [
"sef/renovate-test-scala"
],
"hostRules": [
{
"hostType": "maven",
"endpoint": "https://artifactory.domain.tools/artifactory/domain-release-sbt/",
"username": "username",
"password": "password"
}
],
"packageRules": [
{
"updateTypes": [
"major",
"minor",
"patch"
],
"managers": [
"sbt"
],
"depTypeList": [
"dependencies",
"devDependencies"
],
"automerge": false,
"registryUrls": [" https://artifactory.domain.tools/artifactory/domain-release-sbt/"]
}
],
"sbt": {
"enabled": true
},
"enabledManagers": [
"sbt"
]
}
but I don't see it looking for the private repository
To Reproduce
Additional context
I created a dummy maven project with a single private repository and it worked, it even opened a PR for the outdated private dependency.
Is SBT private repository renovation working?
This needs to be reproduced in a public repository in order to troubleshoot further. Without a reproduction, it won't be possible to answer further.
Remove updateTypes and depTypeList from your first package rule and try again.
If that doesn't work, try to reproduce it as closely as possible in a public github repo (e.g. using the fake artifactory URL as above)
Thank you for the quick response.
I added a testing repository:
stand-alone renovate app with the config
sbt project
Removing updateTypes and depTypeList got me a step forward, I do see renovate looking for the private repositories, but I get unauthorized when trying to access the private artifactory. (it worked before with maven, having the same config in the hostRules).
I also dropped the enabledManagers and only kept "sbt": { "enabled": true }
Now the config looks like this:
{
"endpoint": "https://bitbucket.domain.tools",
"platform": "bitbucket-server",
"username": "username",
"password": "password",
"extends": [
"config:base"
],
"repositories": [
"sef/renovate-test-scala"
],
"hostRules": [
{
"hostType": "maven",
"endpoint": "https://artifactory.domain.tools/artifactory/domain-release-sbt/",
"username": "username",
"password": "password"
}
],
"packageRules": [
{
"managers": [
"sbt"
],
"automerge": false,
"registryUrls": [" https://artifactory.domain.tools/artifactory/domain-release-sbt/"]
}
],
"sbt": {
"enabled": true
}
}
debug message:
Dependency lookup unauthorized. Please add authentication with a hostRule (repository=sef/renovate-test-scala)
"failedUrl": " https://artifactory.domain.tools/artifactory/domain-release-sbt/com/domain/"
I see it stange that is trying to look for the same path but the packages are split by a dot(.) and not by a slash(/)
Dependency lookup unauthorized. Please add authentication with a hostRule (repository=sef/renovate-test-scala) "failedUrl": " https://artifactory.domain.tools/artifactory/domain-release-sbt/com.domain/"
This repo does not have the config you are talking about: https://github.com/marius-carp/renovate-test-scala/blob/main/renovate.json. We need a reproduction
The config is under https://github.com/marius-carp/tool-renovate/blob/main/config.json
Please copy those rules across into the renovate.json of your reproduction repo. The aim is that someone can run something like renovate marius-carp/renovate-test-scala --dry-run and reproduce the lookup.
Can you also clarify what is the url you think Renovate should be looking up, versus what it is looking up? Maybe the problem is that the URL is wrong, but that triggers a 403 from your server.
I pushed a commit with the configs, please have a look to see if is what is supposed to be.
Where maven looks up for the private repositories:
Looking up com.domain: lib-accounts_2.13 in repository "https://artifactory.domain.tools/artifactory/domain-release-sbt/"
Sbt error:
DEBUG: No versions found for com.domain:lib-accounts_2.13 in 2 repositories (repository=sef/renovate-test-scala)
DEBUG: Dependency lookup unauthorized. Please add authentication with a hostRule (repository=sef/renovate-test-scala)
"failedUrl": "https://artifactory.domain.tools/artifactory/domain-release-sbt/com/domain/"
DEBUG: Dependency lookup unauthorized. Please add authentication with a hostRule (repository=sef/renovate-test-scala)
"failedUrl": "https://artifactory.domain.tools/artifactory/domain-release-sbt/com.domain/"
Please fix the reproduction repo so that a dry run can be run against it. It should just contain repository config options (e.g. hostRules, packageRules).
INFO: Found renovate config errors (repository=marius-carp/renovate-test-scala)
"errors": [
{
"depName": "Configuration Error",
"message": "Invalid configuration option: user"
}
]
INFO: Repository has invalid config (repository=marius-carp/renovate-test-scala)
"error": {
"configFile": "renovate.json",
"validationError": "The renovate configuration file contains some invalid settings",
"validationMessage": "Invalid configuration option: user",
"message": "config-validation",
"stack": "Error: config-validation\n at mergeRenovateConfig (/Users/rhys/src/renovate/lib/workers/repository/init/config.ts:164:19)\n at async getRepoConfig (/Users/rhys/src/renovate/lib/workers/repository/init/config.ts:238:12)\n at async initRepo (/Users/rhys/src/renovate/lib/workers/repository/init/index.ts:21:12)\n at async Object.renovateRepository (/Users/rhys/src/renovate/lib/workers/repository/index.ts:37:14)\n at async Object.start (/Users/rhys/src/renovate/lib/workers/global/index.ts:67:7)\n at async /Users/rhys/src/renovate/lib/renovate.ts:10:22"
}
To clarify: we want to work out if it's a wrong URL problem, or a wrong authentication problem.
Sorry for that, I should have rechecked the config before pushing it. I fixed it now.
It looks like the authentication to the private artifactory is not working in sbt.
There are three dependencies extracted whereas ideally we'd have 1 for a minimal reproduction. Here's the debug logs:
INFO: Dependency extraction complete (repository=marius-carp/renovate-test-scala)
"baseBranch": "main",
"stats": {
"managers": {"sbt": {"fileCount": 1, "depCount": 3}},
"total": {"fileCount": 1, "depCount": 3}
}
DEBUG: Looking up org.scala-lang:scala-library in repository https://artifactory.domain.tools/artifactory/domain-release-sbt/ (repository=marius-carp/renovate-test-scala)
DEBUG: Connection refused to maven registry (repository=marius-carp/renovate-test-scala)
"failedUrl": "https://artifactory.domain.tools/artifactory/domain-release-sbt/org/scala-lang/scala-library/maven-metadata.xml"
DEBUG: Content is not found for Maven url: https://artifactory.domain.tools/artifactory/domain-release-sbt/org/scala-lang/scala-library/maven-metadata.xml (repository=marius-carp/renovate-test-scala)
DEBUG: Failed to look up dependency scala (repository=marius-carp/renovate-test-scala, packageFile=build.sbt, dependency=scala)
DEBUG: Connection refused to maven registry (repository=marius-carp/renovate-test-scala)
"failedUrl": " https://artifactory.domain.tools/artifactory/domain-release-sbt/com/domain/"
DEBUG: Connection refused to maven registry (repository=marius-carp/renovate-test-scala)
"failedUrl": " https://artifactory.domain.tools/artifactory/domain-release-sbt/com/typesafe/akka/"
DEBUG: Connection refused to maven registry (repository=marius-carp/renovate-test-scala)
"failedUrl": " https://artifactory.domain.tools/artifactory/domain-release-sbt/com.domain/"
DEBUG: No versions found for com.domain:lib-accounts_2.13 in 2 repositories (repository=marius-carp/renovate-test-scala)
DEBUG: Failed to look up dependency com.domain:lib-accounts (repository=marius-carp/renovate-test-scala, packageFile=build.sbt, dependency=com.domain:lib-accounts)
DEBUG: Connection refused to maven registry (repository=marius-carp/renovate-test-scala)
"failedUrl": " https://artifactory.domain.tools/artifactory/domain-release-sbt/com.typesafe.akka/"
DEBUG: No versions found for com.typesafe.akka:akka-actor_2.13 in 2 repositories (repository=marius-carp/renovate-test-scala)
DEBUG: Failed to look up dependency com.typesafe.akka:akka-actor (repository=marius-carp/renovate-test-scala, packageFile=build.sbt, dependency=com.typesafe.akka:akka-actor)
DEBUG: Package releases lookups complete (repository=marius-carp/renovate-test-scala)
"baseBranch": "main"
DEBUG: packageFiles with updates (repository=marius-carp/renovate-test-scala)
"config": {
"sbt": [
{
"packageFile": "build.sbt",
"manager": "sbt",
"deps": [
{
"registryUrls": [
"https://repo.maven.apache.org/maven2",
"https://artifactory.domain.tools/artifactory/domain-release-sbt"
],
"datasource": "maven",
"depName": "scala",
"lookupName": "org.scala-lang:scala-library",
"currentValue": "2.13.3",
"separateMinorPatch": true,
"depIndex": 0,
"updates": [],
"warnings": [
{
"depName": "scala",
"message": "Failed to look up dependency scala"
}
],
"fixedVersion": "2.13.3"
},
{
"registryUrls": [
"https://repo.maven.apache.org/maven2",
"https://artifactory.domain.tools/artifactory/domain-release-sbt"
],
"depName": "com.domain:lib-accounts",
"lookupName": "com.domain:lib-accounts_2.13",
"currentValue": "15.0.0",
"datasource": "sbt-package",
"depIndex": 1,
"updates": [],
"warnings": [
{
"depName": "com.domain:lib-accounts",
"message": "Failed to look up dependency com.domain:lib-accounts"
}
],
"fixedVersion": "15.0.0"
},
{
"registryUrls": [
"https://repo.maven.apache.org/maven2",
"https://artifactory.domain.tools/artifactory/domain-release-sbt"
],
"depName": "com.typesafe.akka:akka-actor",
"lookupName": "com.typesafe.akka:akka-actor_2.13",
"currentValue": "2.6.9",
"datasource": "sbt-package",
"depIndex": 2,
"updates": [],
"warnings": [
{
"depName": "com.typesafe.akka:akka-actor",
"message": "Failed to look up dependency com.typesafe.akka:akka-actor"
}
],
"fixedVersion": "2.6.9"
}
]
}
]
}
Which of those 3 should we focus on?
This one:
{
"registryUrls": [
"https://repo.maven.apache.org/maven2",
"https://artifactory.domain.tools/artifactory/domain-release-sbt"
],
"depName": "com.domain:lib-accounts",
"lookupName": "com.domain:lib-accounts_2.13",
"currentValue": "15.0.0",
"datasource": "sbt-package",
"depIndex": 1,
"updates": [],
"warnings": [
{
"depName": "com.domain:lib-accounts",
"message": "Failed to look up dependency com.domain:lib-accounts"
}
],
"fixedVersion": "15.0.0"
}
Is it possible to add the same dependency but for maven package manager (pom.xml) so we can compare the URLs?
It looks like we tried two URLs:
DEBUG: Connection refused to maven registry (repository=marius-carp/renovate-test-scala)
"failedUrl": " https://artifactory.domain.tools/artifactory/domain-release-sbt/com/domain/"
DEBUG: Connection refused to maven registry (repository=marius-carp/renovate-test-scala)
"failedUrl": " https://artifactory.domain.tools/artifactory/domain-release-sbt/com.domain/"
In this case the connection refused is expected because this is a dummy URL, but in your real case we'd hope one of those URLs are correct and return a response
I have updated the code to point to a private dummy artifactory. I added the username and password in the config. You can check now the debug logs. Those are the errors that I see.
I just noticed one thing that needs changing: remove the hostType setting from your host rule. The scala packages have "datasource": "sbt-package" which means they'll be looked up with hostType=sbt-package and not maven.
Ok, removing hostType from hostRules did the job, it works on github. But going back to bitbucket is still not working. I get the same "Dependency lookup unauthorized. Please add authentication with a hostRule", is there something else that I need to add to make it work on bitbucket.
- bitbucket is self-hosted
- jfrog artifactory is self-hosted
The core logic of sbt, datasources, and host or package rules are all common between platforms. Any difference is likely because of how you're configuring it (bot vs repository, etc).
It worked, removing managers from under packageRules did it.
The final config version looks like this:
{
"endpoint": "https://bitbucket.domain.tools",
"platform": "bitbucket-server",
"username": "username",
"password": "password",
"extends": [
"config:base"
],
"repositories": [
"sef/renovate-test-scala"
],
"hostRules": [
{
"endpoint": "https://artifactory.domain.tools/artifactory/domain-release-sbt/",
"username": "username",
"password": "password"
}
],
"packageRules": [
{
"automerge": false,
"registryUrls": [" https://artifactory.domain.tools/artifactory/domain-release-sbt/"]
}
],
"sbt": {
"enabled": true
}
}
Thank you a a lot @rarkins for your patience and help. You are a really nice guy!
I'll write a blog post on how to setup renovate for sbt for private repos on seflhosted artifactory.
Sounds great! Congratulations on getting it up and running