GitHub Personal Access token getting invalidated

Question

GitHub Personal Access token getting invalidated

tburow opened this issue 4 years ago · comments

we are using this plugin along with the Github Organization jobs to accomplish automated PR builds.

the issue we are having is that somewhere in the communication with this configuration the Github PATs is getting invalidated.
we have a simple service account with a PAT token on Github Enterprise set up for processing Pull requests.

periodically when trying to upload coverage reports

[GitHub PR Status] start
[GitHub PR Status] getting master coverage...
use default coverage repo

results in

GitHub has been notified of this commit’s build result

java.io.IOException: Server returned HTTP response code: 401 for URL: https://github.digitalglobe.com/api/v3/user
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1900)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)
	at org.kohsuke.github.GitHubHttpUrlConnectionClient.getResponseInfo(GitHubHttpUrlConnectionClient.java:69)
	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:400)
Caused: java.io.IOException: Server returned HTTP response code: 401 for URL: https://github.digitalglobe.com/api/v3/user
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1950)
	at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1945)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1944)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1514)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268)
	at org.kohsuke.github.GitHubHttpUrlConnectionClient$HttpURLConnectionResponseInfo.bodyStream(GitHubHttpUrlConnectionClient.java:197)
	at org.kohsuke.github.GitHubResponse$ResponseInfo.getBodyAsString(GitHubResponse.java:326)
	at org.kohsuke.github.GitHubResponse.parseBody(GitHubResponse.java:91)
	at org.kohsuke.github.GitHubClient.lambda$fetch$0(GitHubClient.java:145)
	at org.kohsuke.github.GitHubClient.createResponse(GitHubClient.java:461)
	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:412)
Caused: org.kohsuke.github.HttpException: {"message":"Bad credentials","documentation_url":"https://docs.github.com/enterprise/2.22/rest"}
	at org.kohsuke.github.GitHubClient.interpretApiError(GitHubClient.java:495)
	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:420)
	at org.kohsuke.github.GitHubClient.fetch(GitHubClient.java:145)
	at org.kohsuke.github.GitHubClient.<init>(GitHubClient.java:134)
	at org.kohsuke.github.GitHubHttpUrlConnectionClient.<init>(GitHubHttpUrlConnectionClient.java:45)
	at org.kohsuke.github.GitHub.<init>(GitHub.java:115)
	at org.kohsuke.github.GitHubBuilder.build(GitHubBuilder.java:423)
	at org.kohsuke.github.GitHub.connectToEnterpriseWithOAuth(GitHub.java:178)
	at org.kohsuke.github.GitHub.connectToEnterprise(GitHub.java:157)
	at com.github.terma.jenkins.githubprcoveragestatus.GitHubPullRequestRepository.getGitHub(GitHubPullRequestRepository.java:68)
	at com.github.terma.jenkins.githubprcoveragestatus.GitHubPullRequestRepository.getGitHubRepository(GitHubPullRequestRepository.java:39)
	at com.github.terma.jenkins.githubprcoveragestatus.CompareCoverageAction.perform(CompareCoverageAction.java:137)
	at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:123)
	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:99)
	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:69)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

The Declaritive pipeline looks like this

        stage('PR Coverage to Github') {
            when { allOf {not { branch 'master' }; expression { return env.CHANGE_ID != null }} }
            steps {
                script {
                    currentBuild.result = 'SUCCESS'
                 }
                step([$class: 'CompareCoverageAction',
                              publishResultAs: 'statusCheck',
                              scmVars: [GIT_URL: '$GIT_URL']])
            }
        }

Digging around Github logs hasnt produced anything useful as of yet.

tburow commented 4 years ago

NICE!

tburow · Answer 1 · Tue Nov 24 2020 02:41:01 GMT+0800 (China Standard Time)

My suspicion is when the Coverage check fails (drop in coverage) - thats when the token is getting invalidated.

tburow · Answer 2 · Tue Nov 24 2020 03:04:20 GMT+0800 (China Standard Time)

another Error output from this plugin -

java.lang.UnsupportedOperationException: no public field ‘sonarLogin’ (or getter method) found in class com.github.terma.jenkins.githubprcoveragestatus.CompareCoverageAction

akshaygupta2208 · Answer 3 · Fri Nov 27 2020 17:57:07 GMT+0800 (China Standard Time)

I am also facing the same error. On the master branch, I am able to push the report but on other branches, it is failing with the same error.

singhmanpreet2 · Answer 4 · Fri Nov 27 2020 18:04:13 GMT+0800 (China Standard Time)

I am facing the same issue as well. I am able to fetch existing code coverage on master branch to the pipeline but while pushing the coverage for feature branch to Pull request it fails.

Error: hudson.remoting.ProxyException: java.io.IOException: Server returned HTTP response code: 401 for

Any updates on this ?

Laurent Dormoy · Answer 5 · Mon Nov 30 2020 17:41:39 GMT+0800 (China Standard Time)

Also having this issue.

My suspicion is when the Coverage check fails (drop in coverage) - thats when the token is getting invalidated.

I tried to reproduce that but can't validate it. The plugin still works after I removed some unit tests from a dummy PR and ran the Jenkins job twice. although the corresponding GitHub check reports a too low coverage.

UPDATE: it fails again now. Looks like I needed to open a new PR in a new project.

In my case this error started to happen during the migration of our Jenkins from an EC2 instance with in-place updates to a containerized Jenkins master built from scratch every deployment (using JCaSC, although this plugin itself doesn't support JCaSC, so it is manually configured).

At some point the token stops working, but not at every deployment. What seems to happen is that the token saved on disk gets somehow overwritten with a non-valid value. Manually changing it back to the original value fixes the issue for some time. Still trying to understand what exactly causes this change of value...

tburow · Answer 6 · Tue Dec 01 2020 03:45:45 GMT+0800 (China Standard Time)

Ive run my self in circles on this issue - I honestly don't know whats causing it - but - My Recommendation for the plugin is to update it use the credential store for pulling the PAT vrs storing it in the plugin config. doing so should eliminate the issue, albeit indirectly.

the one thing I can say for certain is that the issue is isolated to this plugin - through various troubleshooting and reconfig - isolating credentials ect. - using the PAT in ONLY this plugin - the issue still reproduces - tho seemingly random at this point.

tburow · Answer 7 · Tue Dec 01 2020 03:49:35 GMT+0800 (China Standard Time)

another thought on this is that the plugin needs the Github API updated in it. it is out of date. - remotely possible that could have affect. however I dont have the logging to validate that.

Adnan Alam · Answer 8 · Thu Feb 04 2021 04:47:54 GMT+0800 (China Standard Time)

@tburow we faced similar issue after updating our Jenkins version from 2.163.1 to 2.263.1. Please look at the pr #94 when you get a chance. We have this built and deployed on our Jenkins node.
Main issue is that your Jenkins version does not match the one that the plugin is configured for. So please update the jenkins.version in pom to match your jenkins version and that should fix your issue.

tburow · Answer 9 · Thu Feb 04 2021 05:33:04 GMT+0800 (China Standard Time)

@tburow we faced similar issue after updating our Jenkins version from 2.163.1 to 2.263.1. Please look at the pr #94 when you get a chance. We have this built and deployed on our Jenkins node.
Main issue is that your Jenkins version does not match the one that the plugin is configured for. So please update the jenkins.version in pom to match your jenkins version and that should fix your issue.
@adnancerner
im not finding the 4.16 plugin dependency when I try to build it?

Adnan Alam · Answer 10 · Thu Feb 04 2021 06:14:57 GMT+0800 (China Standard Time)

@tburow we faced similar issue after updating our Jenkins version from 2.163.1 to 2.263.1. Please look at the pr #94 when you get a chance. We have this built and deployed on our Jenkins node.
Main issue is that your Jenkins version does not match the one that the plugin is configured for. So please update the jenkins.version in pom to match your jenkins version and that should fix your issue.
@adnancerner
im not finding the 4.16 plugin dependency when I try to build it?

This is the latest release I found. When I build it locally with

mvn clean install hpi:hpi

I don't run into any issues. Also possible that if you have an older jenkins version then 4.16 maybe is not compatible so maybe use 4.11? I would try to downgrade till works maybe?
I also ran

mvn hpi:run -Djetty.port=1900

and verified that the jenkins instance on that port had the plugin installed properly.

tburow · Answer 11 · Thu Feb 04 2021 07:09:29 GMT+0800 (China Standard Time)

@adnancerner im sure its my environment - I have not compiled Jenkins plugins directly before - fumbling through it I finally got it to compile

Mockito fails for me - but skipping test it compiles fine. again I think thats probably my env. I had another developer run the build and it went fine.

Adnan Alam · Answer 12 · Thu Feb 04 2021 08:33:24 GMT+0800 (China Standard Time)

@adnancerner im sure its my environment - I have not compiled Jenkins plugins directly before - fumbling through it I finally got it to compile

Mockito fails for me - but skipping test it compiles fine. again I think thats probably my env. I had another developer run the build and it went fine.

Glad it is working for you! Let me know if you face any issues as that'll help us prepare for them as well.

Adnan Alam · Answer 13 · Thu Feb 11 2021 04:10:56 GMT+0800 (China Standard Time)

Here's the actual issue that I found:
2021-02-10 18:49:08.305+0000 [id=64] WARNING hudson.Functions#getPasswordValue: <f:password/> form control in Jenkins/configure.jelly is not backed by hudson.util.Secret. Learn more: https://jenkins.io/redirect/hudson.util.Secret
Basically in the global.groovy in the resource folder the password tag calls the method getPasswordValue which fails to retrieve the password due to some update in Secret util so just need to decrypt that value when the form is submitted.

Reproduce:

make any change that is not the PAT, then you'll see the PAT has been overwritten with the encrypted version.

The latest update to my PR should definitely fix the issue 😅

This is the specific commit that addresses the main problem (PAT getting overwritten with encrypted version).