WinRM Error: java.net.SocketTimeoutException: connect timed out
pts-aut opened this issue · comments
We are using rundeck with overthere-winrm plugin and the default executors are set:
"node-executor": "overthere-winrm"
"file-copier": "overthere-winrm"
Also we have set the timeout parameters to a high value:
framework.winrm-timeout=PT1H
framework.winrm-connection-timeout=28800000
But in some executions we are getting an "connect timed out" error from the FileCopier - a retry is running without any problem.
Is there another parameter separately to winrm-connection-timeout for the FileCopier? The defined timeout values are not reached by far.
com.dtolabs.rundeck.core.execution.workflow.steps.node.NodeStepException: com.dtolabs.rundeck.core.execution.service.FileCopierException: WinRM Error: java.net.SocketTimeoutException: connect timed out
at com.dtolabs.rundeck.plugin.copyfile.CopyFileNodeStepPlugin.copyFileList(CopyFileNodeStepPlugin.java:221) ~[?:?]
at com.dtolabs.rundeck.plugin.copyfile.CopyFileNodeStepPlugin.executeNodeStep(CopyFileNodeStepPlugin.java:113) ~[?:?]
at com.dtolabs.rundeck.core.execution.workflow.steps.node.NodeStepPluginAdapter.executeNodeStep(NodeStepPluginAdapter.java:167) ~[rundeck-core-4.8.0-20221110.jar!/:?]
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.executeNodeStep(ExecutionServiceImpl.java:206) ~[rundeck-core-4.8.0-20221110.jar!/:?]
at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:130) ~[rundeck-core-4.8.0-20221110.jar!/:?]
at com.dtolabs.rundeck.core.execution.dispatch.SequentialNodeDispatcher.dispatch(SequentialNodeDispatcher.java:61) ~[rundeck-core-4.8.0-20221110.jar!/:?]
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.dispatchToNodesWith(ExecutionServiceImpl.java:262) ~[rundeck-core-4.8.0-20221110.jar!/:?]
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.dispatchToNodes(ExecutionServiceImpl.java:233) ~[rundeck-core-4.8.0-20221110.jar!/:?]
at com.dtolabs.rundeck.core.execution.workflow.steps.NodeDispatchStepExecutor.executeWorkflowStep(NodeDispatchStepExecutor.java:66) ~[rundeck-core-4.8.0-20221110.jar!/:?]
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.executeStep(ExecutionServiceImpl.java:111) ~[rundeck-core-4.8.0-20221110.jar!/:?]
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowExecutor.executeWFItem(BaseWorkflowExecutor.java:285) ~[rundeck-core-4.8.0-20221110.jar!/:?]
at com.dtolabs.rundeck.core.execution.workflow.BaseWorkflowExecutor.executeWorkflowStep(BaseWorkflowExecutor.java:681) ~[rundeck-core-4.8.0-20221110.jar!/:?]
at com.dtolabs.rundeck.core.execution.workflow.engine.StepCallable.apply(StepCallable.java:71) ~[rundeck-core-4.8.0-20221110.jar!/:?]
at com.dtolabs.rundeck.core.execution.workflow.engine.StepOperation.apply(StepOperation.java:76) ~[rundeck-core-4.8.0-20221110.jar!/:?]
at com.dtolabs.rundeck.core.execution.workflow.engine.StepOperation.apply(StepOperation.java:32) ~[rundeck-core-4.8.0-20221110.jar!/:?]
at com.dtolabs.rundeck.core.rules.WorkflowEngineOperationsProcessor.lambda$beginOperation$1(WorkflowEngineOperationsProcessor.java:323) ~[rundeck-core-4.8.0-20221110.jar!/:?]
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) [guava-26.0-jre.jar!/:?]
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) [guava-26.0-jre.jar!/:?]
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) [guava-26.0-jre.jar!/:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: com.dtolabs.rundeck.core.execution.service.FileCopierException: WinRM Error: java.net.SocketTimeoutException: connect timed out
at com.dtolabs.rundeck.plugin.overthere.OTWinRMFileCopier.copyFileStream(OTWinRMFileCopier.java:129) ~[?:?]
at com.dtolabs.rundeck.plugin.overthere.OTWinRMFileCopier.copyFile(OTWinRMFileCopier.java:139) ~[?:?]
at com.dtolabs.rundeck.core.execution.service.MultiFileCopierUtil.copyMultipleFiles(MultiFileCopierUtil.java:62) ~[rundeck-core-4.8.0-20221110.jar!/:?]
at com.dtolabs.rundeck.core.execution.ExecutionServiceImpl.fileCopyFiles(ExecutionServiceImpl.java:352) ~[rundeck-core-4.8.0-20221110.jar!/:?]
at com.dtolabs.rundeck.plugin.copyfile.CopyFileNodeStepPlugin.copyFileList(CopyFileNodeStepPlugin.java:207) ~[?:?]
... 21 more
Also the winrm settings on the destination host are set to high timeouts:
winrm get winrm/config
Config
MaxEnvelopeSizekb = 4294967295
MaxTimeoutms = 3600000
MaxBatchItems = 4294967295
MaxProviderRequests = 4294967295
Client
NetworkDelayms = 5000
URLPrefix = wsman
AllowUnencrypted = false
Auth
Basic = true
Digest = true
Kerberos = true
Negotiate = true
Certificate = true
CredSSP = false
DefaultPorts
HTTP = 5985
HTTPS = 5986
TrustedHosts
Service
RootSDDL = O:NSG:BAD:P(A;;GA;;;BA)(A;;GR;;;IU)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)
MaxConcurrentOperations = 4294967295
MaxConcurrentOperationsPerUser = 4294967295
EnumerationTimeoutms = 4294967295
MaxConnections = 300
MaxPacketRetrievalTimeSeconds = 36000
AllowUnencrypted = true
Auth
Basic = true
Kerberos = true
Negotiate = true
Certificate = false
CredSSP = false
CbtHardeningLevel = Relaxed
DefaultPorts
HTTP = 5985
HTTPS = 5986
IPv4Filter = *
IPv6Filter = *
EnableCompatibilityHttpListener = false
EnableCompatibilityHttpsListener = false
CertificateThumbprint
AllowRemoteAccess = true
Winrs
AllowRemoteShellAccess = true
IdleTimeout = 2147483647
MaxConcurrentUsers = 2147483647
MaxShellRunTime = 2147483647
MaxProcessesPerShell = 2147483647
MaxMemoryPerShellMB = 4096
MaxShellsPerUser = 2147483647
Debug output of a successful execution:
[overthere-winrm:hostname.local]
[overthere-winrm:hostname.local] ConnectionOptions[
connectionTimeoutMillis --> 28800000
winrmEnableHttps --> false
password --> ********
address --> hostname.local
os --> WINDOWS
winrmKerberosUseHttpSpn --> false
port --> 5985
winrmKerberosAddPortToSpn --> false
winrmTimeout --> PT1H
file-copy-destination-dir --> C:/temp
connectionType --> WINRM_INTERNAL
username --> user1
]
[overthere-winrm:hostname.local] Destination file acquired C:\temp\file.ps1
[overthere-winrm:hostname.local] Copying file to C:\temp\file1.ps1
[overthere-winrm:hostname.local] File copied at: C:\temp\file1.ps1
[overthere-winrm:hostname.local] Closing connection
[overthere-winrm:hostname.local]
[overthere-winrm:hostname.local] ConnectionOptions[
connectionTimeoutMillis --> 28800000
winrmEnableHttps --> false
password --> ********
address --> hostname.local
os --> WINDOWS
winrmKerberosUseHttpSpn --> false
port --> 5985
winrmKerberosAddPortToSpn --> false
winrmTimeout --> PT1H
file-copy-destination-dir --> C:/temp
connectionType --> WINRM_INTERNAL
username --> user1
]
Versions:
Rundeck 4.8.0
WinRM Plugin 1.3.8 (using com.xebialabs.overthere 5.0.2)