TFS/VSTS Package Management authentication
Megasware128 opened this issue · comments
Great to use Yarn as a build task but unlike the npm tasks available, Yarn doesn't seem to automatically authenticate for npm registries created by the Package Management feature in TFS2017/VSTS. Is there a solution for using Package Management registries in combination with the Yarn build task?
This is of course a great suggestion. We need to investigate how we can adapt the authentication procedure from the npm task to yarn https://github.com/Microsoft/vsts-tasks/blob/master/Tasks/Npm/npmtask.ts#L190 .
This is a priority one for us.
Looks like the code you want to be looking at is in https://github.com/Microsoft/vsts-tasks/blob/master/Tasks/Npm/npmregistry.ts
We can't switch to yarn on our build server because of this :(
Happy to accept a PR is anyone has the time to work on this.
@sandorfr Would have loved to work on this. But I lack the context.
I might take a look but need to find some time
A manual way to set the access token using a build agent that we use at the moment is the following PowerShell script:
Add-Content <path-to-your-npmrc> "`r`n//<your-team>.pkgs.visualstudio.com/_packaging/<your-feed-name>/npm/registry/:_authToken=$(System.AccessToken)"
It will add the access token to the .npmrc so yarn can use it to access the registry. You should run this before doing yarn or yarn install. Yarn will consider any .npmrc files, even upwards in the folder tree, so just put it somewhere within the tree and it should work.
Please mind: this requires to have the following Build Definition Option to be set:
I adjusted the roadmap, as I'm focused on this issue right now. This is basically around the corner, but I'm running into issues similar to this yarnpkg/yarn#678 . It seems like it might be necessary to generate a yarnrc in addition to npmrc.
I've pushed a new major version including VSTS feed support.
Broken for me. npm install works fined, yarn install yields 401 with VSTS feeds.
Also getting lots of warnings:
2017-09-13T10:15:02.0212231Z ##[warning]Resource file haven\'t set, can\'t find loc string for key: FoundLocalRegistries
2017-09-13T10:15:02.0222238Z ##[warning]Resource file haven\'t set, can\'t find loc string for key: FoundBuildCredentials
2017-09-13T10:15:03.4391547Z ##[warning]Resource file haven\'t set, can\'t find loc string for key: SavingFile
I have tried with Registries in my .npmrc file and Registry I select here, both give 401. Either of these options work fine with npm task.
Using @cryza's method works fine.
@sandorfr weird, it's working now. I was having issues with VSTS feeds locally too so perhaps this was unrelated.
Ok it's still broken on my build. Here's debug log
******************************************************************************
Starting: Yarn install --verbose
******************************************************************************
==============================================================================
Task : Yarn task
Description : Executes Yarn
Version : 1.1.1001
Author : Geek Learning
Help : [More Information](https://github.com/geeklearningio/gl-vsts-tasks-yarn/wiki/Yarn) (Version 1.1.1001)
==============================================================================
agent.workFolder=d:\a
loading inputs and endpoints
loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
loading INPUT_ARGUMENTS
loading INPUT_CUSTOMFEED
loading INPUT_CUSTOMREGISTRY
loading INPUT_PRODUCTIONMODE
loading INPUT_PROJECTDIRECTORY
loading SECRET_NPM_ACCESSTOKEN
loaded 9
which 'yarn'
found: 'C:\NPM\Modules\yarn.cmd'
Arguments=install --verbose
ProjectDirectory=d:\a\1\s
customRegistry=useNpmrc
customFeed=c314204c-5eb4-4dbd-b660-e0c97452ddd5
customEndpoint=null
C:\NPM\Modules\yarn.cmd
Build.BuildId=608
Agent.BuildDirectory=d:\a\1
testing directory 'd:\a\1\npm'
testing directory 'd:\a\1'
mkdir 'd:\a\1\npm'
Build.BuildId=608
Agent.BuildDirectory=d:\a\1
Npm.PackagingCollectionUrl=undefined
System.TeamFoundationCollectionUri=https://{companyName}.visualstudio.com/
Resource file haven\'t set, can\'t find loc string for key: FoundLocalRegistries
Processed: ##vso[task.issue type=warning;]Resource file haven\'t set, can\'t find loc string for key: FoundLocalRegistries
FoundLocalRegistries 1
SYSTEMVSSCONNECTION exists true
SYSTEMVSSCONNECTION exists true
Resource file haven\'t set, can\'t find loc string for key: FoundBuildCredentials
Processed: ##vso[task.issue type=warning;]Resource file haven\'t set, can\'t find loc string for key: FoundBuildCredentials
FoundBuildCredentials
Npm.PackagingCollectionUrl=undefined
System.TeamFoundationCollectionUri=https://{companyName}.visualstudio.com/
SYSTEMVSSCONNECTION exists true
SYSTEMVSSCONNECTION exists true
Resource file haven\'t set, can\'t find loc string for key: FoundBuildCredentials
Processed: ##vso[task.issue type=warning;]Resource file haven\'t set, can\'t find loc string for key: FoundBuildCredentials
FoundBuildCredentials
Using registries in .npmrc
customEndpoint=null
Adding auth for registry: https://{companyName}.pkgs.visualstudio.com/_packaging/{FeedName}/npm/registry/
which 'C:\NPM\Modules\yarn.cmd'
found: 'C:\NPM\Modules\yarn.cmd'
which 'C:\NPM\Modules\yarn.cmd'
found: 'C:\NPM\Modules\yarn.cmd'
ProductionMode=false
C:\NPM\Modules\yarn.cmd arg: install --verbose
exec tool: C:\NPM\Modules\yarn.cmd
arguments:
install
--verbose
C:\Windows\system32\cmd.exe /D /S /C "C:\NPM\Modules\yarn.cmd install --verbose"
yarn install v1.0.1
verbose 0.684 Checking for configuration file "d:\\a\\1\\s\\.npmrc".
verbose 0.685 Found configuration file "d:\\a\\1\\s\\.npmrc".
verbose 0.685 Checking for configuration file "C:\\Users\\buildguest\\.npmrc".
verbose 0.686 Checking for configuration file "d:\\a\\_tool\\node\\8.5.0\\x64\\etc\\npmrc".
verbose 0.686 Checking for configuration file "d:\\a\\1\\s\\.npmrc".
verbose 0.686 Found configuration file "d:\\a\\1\\s\\.npmrc".
verbose 0.687 Checking for configuration file "d:\\a\\1\\.npmrc".
verbose 0.687 Checking for configuration file "d:\\a\\.npmrc".
verbose 0.691 Checking for configuration file "d:\\a\\1\\s\\.yarnrc".
verbose 0.691 Checking for configuration file "C:\\Users\\buildguest\\.yarnrc".
verbose 0.692 Checking for configuration file "d:\\a\\_tool\\node\\8.5.0\\x64\\etc\\yarnrc".
verbose 0.692 Checking for configuration file "d:\\a\\1\\s\\.yarnrc".
verbose 0.692 Checking for configuration file "d:\\a\\1\\.yarnrc".
verbose 0.692 Checking for configuration file "d:\\a\\.yarnrc".
verbose 0.696 current time: 2017-09-18T15:49:42.957Z
Resolving packages...
Fetching packages...
verbose 2.343 Performing "GET" request to "https://{companyName}.pkgs.visualstudio.com/_packaging/{FeedName}/npm/registry/axios/-/axios-0.16.2.tgz".
verbose 2.356 Performing "GET" request to "https://{companyName}.pkgs.visualstudio.com/_packaging/{FeedName}/npm/registry/follow-redirects/-/follow-redirects-1.2.4.tgz".
verbose 2.388 Performing "GET" request to "https://{companyName}.pkgs.visualstudio.com/_packaging/{FeedName}/npm/registry/is-buffer/-/is-buffer-1.1.5.tgz".
verbose 2.394 Performing "GET" request to "https://{companyName}.pkgs.visualstudio.com/_packaging/{FeedName}/npm/registry/debug/-/debug-2.6.8.tgz".
verbose 2.396 Performing "GET" request to "https://{companyName}.pkgs.visualstudio.com/_packaging/{FeedName}/npm/registry/ms/-/ms-2.0.0.tgz".
verbose 2.398 Performing "GET" request to "https://{companyName}.pkgs.visualstudio.com/_packaging/{FeedName}/npm/registry/babel-core/-/babel-core-6.26.0.tgz".
verbose 2.4 Performing "GET" request to "https://{companyName}.pkgs.visualstudio.com/_packaging/{FeedName}/npm/registry/babel-code-frame/-/babel-code-frame-6.26.0.tgz".
verbose 2.403 Performing "GET" request to "https://{companyName}.pkgs.visualstudio.com/_packaging/{FeedName}/npm/registry/babel-generator/-/babel-generator-6.26.0.tgz".
verbose 2.482 Error: https://{companyName}.pkgs.visualstudio.com/_packaging/{FeedName}/npm/registry/follow-redirects/-/follow-redirects-1.2.4.tgz: Request failed "401 Unauthorized"
at Request.<anonymous> (C:\NPM\Modules\node_modules\yarn\lib\cli.js:57662:26)
at emitOne (events.js:115:13)
at Request.emit (events.js:210:7)
at Request.module.exports.Request.onRequestResponse (C:\NPM\Modules\node_modules\yarn\lib\cli.js:118210:10)
at emitOne (events.js:115:13)
at ClientRequest.emit (events.js:210:7)
at HTTPParser.parserOnIncomingClient (_http_client.js:565:21)
at HTTPParser.parserOnHeadersComplete (_http_common.js:116:23)
at TLSSocket.socketOnData (_http_client.js:454:20)
at emitOne (events.js:115:13)
error An unexpected error occurred: "https://{companyName}.pkgs.visualstudio.com/_packaging/{FeedName}/npm/registry/follow-redirects/-/follow-redirects-1.2.4.tgz: Request failed \"401 Unauthorized\"".
info If you think this is a bug, please open a bug report with the information provided in "d:\\a\\1\\s\\yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
Error: C:\NPM\Modules\yarn.cmd failed with return code: 1
rc:1
success:false
Error: C:\NPM\Modules\yarn.cmd failed with return code: 1
task result: Failed
Error: C:\NPM\Modules\yarn.cmd failed with return code: 1
Processed: ##vso[task.issue type=error;]Error: C:\NPM\Modules\yarn.cmd failed with return code: 1
Processed: ##vso[task.complete result=Failed;]Error: C:\NPM\Modules\yarn.cmd failed with return code: 1
Agent.BuildDirectory=d:\a\1
rm -rf d:\a\1\npm
removing directory
******************************************************************************
Finishing: Yarn install --verbose
******************************************************************************
If I add a powershell task before it to do this (with access token as an env var):
Add-Content .npmrc "`r`n//{companyName}.pkgs.visualstudio.com/_packaging/{FeedName}/npm/registry/:_authToken=$(npm.accessToken)"
Add-Content .npmrc "`r`n//{companyName}.pkgs.visualstudio.com/_packaging/{FeedName}/npm/:_authToken=$(npm.accessToken)"
It works fine.
I notice the line
Adding auth for registry: https://{companyName}.pkgs.visualstudio.com/_packaging/{FeedName}/npm/registry/
so it should work correctly? Weird. The only difference I see here is that it's potentially (haven't looked at your code) only adding it for /npm/registry/ and not /npm/ as well, and that the scheme is provided as https://, where as I'm providing it as relative scheme.
can you explain to me how this templated url https://{companyName}.pkgs.visualstudio.com/_packaging/{FeedName}/npm/registry/ is working or point me to some documentation? I'm not familiar with this.
It's not real, I should have said https://foo.pkgs.visualstudio.com/_packaging/bar/npm/registry/
Given your logs it seems you have selected the first option but I'm pretty sure you want the second one :
another possible cause is that you don't have the proper rights defined for your package feed :
see documentation: https://docs.microsoft.com/en-us/vsts/package/feeds/common-identities
Tried both, Registries in my .npmrc file and Registry I select here neither work, stirll get 401 without extra task of putting auth token in to .npmrc 😢
Checked perms on the feed too, they're all correct!
I'd assume the way it automatically inserts auth stuff is lifted from this https://github.com/Microsoft/vsts-tasks/tree/master/Tasks/Npm so I'd imagine it should auth in an identical way. Npm task works fine with either Registries in my .npmrc file and Registry I select here, but this doesn't without the extra step. Weird!
I'd guess it's more likely some misconfiguration at my end, otherwise you'd have others reporting issues too?
It's also strange how it happens on the 10th or so package... Perhaps that's a concurrency thing though (not sure if the requests are performed in parallel?)
What I'll try when I get time, is after the yarn install task (which should modify .npmrc) I shall get it to output the contents .npmrc file and see if it's appended an appropriate token to it.
can you show me your .npmrc so I can try to reproduce your configuration on my test env?
registry=https://foo.pkgs.visualstudio.com/_packaging/bar/npm/registry/
always-auth=true
Just copied and pasted from Connect to Feed>npm in VSTS
There is a new line at the bottom of the file, can't imagine that is what is throwing it off? Haven't got time to test without it at the moment, sorry.
Ok reproduced
Have you seen the the NpmAuthenticate Task, I'm quite tempted to remove the Authentication support from the task.
Happy to hear your feedback about it.
@sandorfr No I haven't! Would make it rather more maintainable for you if auth was done externally by MS's task. I'm happy with that.
@Mardoxx, can you confirm that the issue is fixed for you with the latest version ?
Had a feedback from vsts team and they'd prefer us to keep builtin authentication support for security reasons (because the task cleans up credentials).
@sandorfr Works perfectly! Thanks for your support 😄
Still coming up with those resource warnings, e.g. ##[warning]Resource file haven\'t set, can\'t find loc string for key: FoundLocalRegistries
This is also happening to me... was a solution even found?
It’s long addressed and closed. If you have a problem please file a new detailed issue.
@kdelmonte Try uninstalling and reinstalling the extensions
I was getting the same 401 issues when trying to install from an internal packagefeed. The workaround for me, was to include a .npmrc file and use the built-in npm authenticate task.
All permissions should be alright and OAuth script access is turned on.
Are you sure it's working as intended?
PS. also tried to reinstall the extension without luck
As I said this has been resolved a long time ago. If you have a problem, it is probably something specific please open an issue with the proper logs.