Multiple SSH Exec build steps fails

Question

Multiple SSH Exec build steps fails

jonahadmin opened this issue 8 years ago · comments

There seems to be an issue with com.jcraft.jsch.JSchException, or at least how it's being used.

I have two build separate build steps, that both connect to the same server. The first step executes find, but then any subsequent SSH Exec step results in the exception below.

Based on cursory research, I'm guessing that instead of trying to establish a new connection, it tries to somehow reuse the existing one even though it's already been closed.

[2016-06-23 10:34:38,745]  ERROR -    jetbrains.buildServer.AGENT - com.jcraft.jsch.JSchException: session is down

jetbrains.buildServer.RunBuildException: com.jcraft.jsch.JSchException: session is down
    at jetbrains.buildServer.deployer.agent.ssh.SSHExecProcessAdapter.runProcess(SSHExecProcessAdapter.java:46)
    at jetbrains.buildServer.deployer.agent.SyncBuildProcessAdapter.start(SyncBuildProcessAdapter.java:58)
    at jetbrains.buildServer.agent.impl.buildStages.runnerStages.start.CallRunnerStage.doBuildStage(CallRunnerStage.java:58)
    at jetbrains.buildServer.agent.impl.buildStages.RunnerStagesExecutor$1.callStage(RunnerStagesExecutor.java:25)
    at jetbrains.buildServer.agent.impl.buildStages.RunnerStagesExecutor$1.callStage(RunnerStagesExecutor.java:18)
    at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.callRunStage(StagesExecutor.java:78)
    at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.doStages(StagesExecutor.java:37)
    at jetbrains.buildServer.agent.impl.buildStages.RunnerStagesExecutor.doStages(RunnerStagesExecutor.java:18)
    at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.RunnerContextExecutor.callRunnerStages(RunnerContextExecutor.java:43)
    at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.StepExecutor.processNextStep(StepExecutor.java:25)
    at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.ForEachBuildRunnerStage.executeRunnerStep(ForEachBuildRunnerStage.java:138)
    at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.ForEachBuildRunnerStage.runStep(ForEachBuildRunnerStage.java:123)
    at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.ForEachBuildRunnerStage.executeBuildRunners(ForEachBuildRunnerStage.java:83)
    at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.ForEachBuildRunnerStage.doBuildStage(ForEachBuildRunnerStage.java:44)
    at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:31)
    at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:24)
    at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.callRunStage(StagesExecutor.java:78)
    at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.doStages(StagesExecutor.java:37)
    at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor.doStages(BuildStagesExecutor.java:24)
    at jetbrains.buildServer.agent.impl.BuildRunActionImpl.doStages(BuildRunActionImpl.java:70)
    at jetbrains.buildServer.agent.impl.BuildRunActionImpl.runBuild(BuildRunActionImpl.java:50)
    at jetbrains.buildServer.agent.impl.BuildAgentImpl.doActualBuild(BuildAgentImpl.java:288)
    at jetbrains.buildServer.agent.impl.BuildAgentImpl.access$100(BuildAgentImpl.java:53)
    at jetbrains.buildServer.agent.impl.BuildAgentImpl$1.run(BuildAgentImpl.java:252)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.jcraft.jsch.JSchException: session is down
    at com.jcraft.jsch.Channel.sendChannelOpen(Channel.java:762)
    at com.jcraft.jsch.Channel.connect(Channel.java:151)
    at com.jcraft.jsch.Channel.connect(Channel.java:145)
    at jetbrains.buildServer.deployer.agent.ssh.SSHExecProcessAdapter.executeCommand(SSHExecProcessAdapter.java:78)
    at jetbrains.buildServer.deployer.agent.ssh.SSHExecProcessAdapter.runProcess(SSHExecProcessAdapter.java:42)

jonahadmin · Answer 1 · Thu Jun 23 2016 23:02:18 GMT+0800 (China Standard Time)

I think my interpretation of the problem may be incorrect. The issue is inconsistent. I'll follow up if I have better information.

jonahadmin · Answer 2 · Sat Jun 25 2016 05:15:30 GMT+0800 (China Standard Time)

I am unable to determine any kind of pattern at all. I've tried re-arranging build steps, inserting random build steps in case one step was affecting others, but the problem continues to happen completely randomly. It is happening with both the SSH Exec and the SSHDeployer plugins, and it's always the same error, with no other information.

[2016-06-24 17:07:38,597]  ERROR -    jetbrains.buildServer.AGENT - com.jcraft.jsch.JSchException: session is down

jetbrains.buildServer.RunBuildException: com.jcraft.jsch.JSchException: session is down
    at jetbrains.buildServer.deployer.agent.ssh.scp.ScpProcessAdapter.runProcess(ScpProcessAdapter.java:93)
    at jetbrains.buildServer.deployer.agent.SyncBuildProcessAdapter.start(SyncBuildProcessAdapter.java:58)
    at jetbrains.buildServer.agent.impl.buildStages.runnerStages.start.CallRunnerStage.doBuildStage(CallRunnerStage.java:58)
    at jetbrains.buildServer.agent.impl.buildStages.RunnerStagesExecutor$1.callStage(RunnerStagesExecutor.java:25)
    at jetbrains.buildServer.agent.impl.buildStages.RunnerStagesExecutor$1.callStage(RunnerStagesExecutor.java:18)
    at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.callRunStage(StagesExecutor.java:78)
    at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.doStages(StagesExecutor.java:37)
    at jetbrains.buildServer.agent.impl.buildStages.RunnerStagesExecutor.doStages(RunnerStagesExecutor.java:18)
    at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.RunnerContextExecutor.callRunnerStages(RunnerContextExecutor.java:43)
    at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.StepExecutor.processNextStep(StepExecutor.java:25)
    at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.ForEachBuildRunnerStage.executeRunnerStep(ForEachBuildRunnerStage.java:138)
    at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.ForEachBuildRunnerStage.runStep(ForEachBuildRunnerStage.java:123)
    at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.ForEachBuildRunnerStage.executeBuildRunners(ForEachBuildRunnerStage.java:83)
    at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.ForEachBuildRunnerStage.doBuildStage(ForEachBuildRunnerStage.java:44)
    at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:31)
    at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:24)
    at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.callRunStage(StagesExecutor.java:78)
    at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.doStages(StagesExecutor.java:37)
    at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor.doStages(BuildStagesExecutor.java:24)
    at jetbrains.buildServer.agent.impl.BuildRunActionImpl.doStages(BuildRunActionImpl.java:70)
    at jetbrains.buildServer.agent.impl.BuildRunActionImpl.runBuild(BuildRunActionImpl.java:50)
    at jetbrains.buildServer.agent.impl.BuildAgentImpl.doActualBuild(BuildAgentImpl.java:288)
    at jetbrains.buildServer.agent.impl.BuildAgentImpl.access$100(BuildAgentImpl.java:53)
    at jetbrains.buildServer.agent.impl.BuildAgentImpl$1.run(BuildAgentImpl.java:252)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.jcraft.jsch.JSchException: session is down
    at com.jcraft.jsch.Channel.sendChannelOpen(Channel.java:762)
    at com.jcraft.jsch.Channel.connect(Channel.java:151)
    at com.jcraft.jsch.Channel.connect(Channel.java:145)
    at jetbrains.buildServer.deployer.agent.ssh.scp.ScpProcessAdapter.upload(ScpProcessAdapter.java:121)
    at jetbrains.buildServer.deployer.agent.ssh.scp.ScpProcessAdapter.runProcess(ScpProcessAdapter.java:88)
    ... 24 more

jonahadmin · Answer 3 · Sat Jun 25 2016 06:27:03 GMT+0800 (China Standard Time)

Sorry for the stream of consciousness style reporting, but we may have a new theory that there is something flakey going on with the server itself rather than the plugin. I tried performing similar commands using bonafied ssh, and I was getting similarly strange results including write failures. Apparently RHEL7 prior to 7.2 did have ssh bugs. We've just done a full update and I guess we'll see what happens.

jonahadmin · Answer 4 · Sat Jun 25 2016 06:28:24 GMT+0800 (China Standard Time)

FYI, for anyone reading this thread, the server in question is an Amazon Redhat 7.2 AMI. We just did a yum update, and now we'll have to wait to see if it solved the problem.

Nikita Skvortsov · Answer 5 · Sat Jun 25 2016 14:50:24 GMT+0800 (China Standard Time)

Thank you for detailed follow up.
Please, let us know if upgrading the server helps
25 июня 2016 г. 1:28 пользователь "jonahadmin" notifications@github.com
написал:

FYI, for anyone reading this thread, the server in question is an Amazon
Redhat 7.2 AMI. We just did a yum update, and now we'll have to wait to see
if it solved the problem.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#88 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAg5kyPITAgQGrntx5c-IGJ7YXBR6buNks5qPFoJgaJpZM4I84qa
.

Nikita Skvortsov · Answer 6 · Thu Jun 30 2016 21:39:32 GMT+0800 (China Standard Time)

@jonahadmin so, how did upgrade go? Is it any better now?

jonahadmin · Answer 7 · Thu Jun 30 2016 22:16:11 GMT+0800 (China Standard Time)

Since we did the yum update, the problem appears to have gone away complete. So it appears that the problem all along was that the base RHEL 7.2 AMI has some unexpected yet critical bugs with SSH.

I am in awe that they could create an AMI with such serious problems.