A WinRM timeout causes a HTTP 500 but is not notified to OverthereProcessOutputHandler

[zerikv]

Hello,

With OverThere 2.1.0-beta-1 (from Maven), I open a WinRM connection (CifsConnectionBuilder.CIFS_PROTOCOL) configured with a timeout to 10 secondes:

    options.set(CifsConnectionBuilder.WINRM_TIMEMOUT, "PT10.000S");

With this connection, a command line is requested, to execute a PowerShell which sleeps for 12 secondes.
Therefore the SOAP request falls in timeout.

    connection.execute(processOutHandler, CmdLine.build("powershell.exe", "-nonInteractive", "-command", "Start-Sleep -s 12;"));

I can see in the logs this timeout causes an HTTP response from the WinRM service with a HTTP code 500 (please, see logs below).
But the WinRMRuntimeIOException is catched and no handler of OverthereProcessOutputHandler is called.

I would need to intercept this kind of error.

What do you think about this issue?

Thank you for your advices.

10:57:42.371 [Process output reader for command [powershell.exe -nonInteractive -command "Start-Sleep -s 12;"] on [cifs:winrm://admin@TEST-SEVEN:445:5985]] TRACE c.x.o.c.w.c.ApacheHttpComponentsHttpClientHttpConnector - Header Content-Length: 1347
Exception in thread "Process output reader for command [powershell.exe -nonInteractive -command "Start-Sleep -s 12;"] on [cifs:winrm://admin@TEST-SEVEN:445:5985]" com.xebialabs.overthere.cifs.winrm.exception.WinRMRuntimeIOException: Send message on http://TEST-SEVEN:5985/wsman error , document in <?xml version="1.0" encoding="UTF-8"?>

<env:Envelope xmlns:env="http://www.w3.org/2003/05/soap-envelope">
  <env:Header>
    <a:To xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing">http://TEST-SEVEN:5985/wsman</a:To>
    <a:ReplyTo xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing">
      <a:Address mustUnderstand="true">http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous</a:Address>
    </a:ReplyTo>
    <w:MaxEnvelopeSize xmlns:w="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd" mustUnderstand="true">153600</w:MaxEnvelopeSize>
    <a:MessageID xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing">uuid:77193F42-36B1-413F-B425-4F1BEB25E1DC</a:MessageID>
    <w:Locale xmlns:w="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd" mustUnderstand="false" xml:lang="en-US"/>
    <p:DataLocale xmlns:p="http://schemas.microsoft.com/wbem/wsman/1/wsman.xsd" mustUnderstand="false" xml:lang="en-US"/>
    <w:OperationTimeout xmlns:w="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd">PT10.000S</w:OperationTimeout>
    <a:Action xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing" mustUnderstand="true">http://schemas.microsoft.com/wbem/wsman/1/windows/shell/Receive</a:Action>
    <w:SelectorSet xmlns:w="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd">
      <w:Selector Name="ShellId">61B4EFC1-35B5-45DF-BB87-8CE4E757C54D</w:Selector>
    </w:SelectorSet>
    <w:ResourceURI xmlns:w="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd" mustUnderstand="true">http://schemas.microsoft.com/wbem/wsman/1/windows/shell/cmd</w:ResourceURI>
  </env:Header>
  <env:Body>
    <rsp:Receive xmlns:rsp="http://schemas.microsoft.com/wbem/wsman/1/windows/shell">
      <rsp:DesiredStream CommandId="3393F071-7BC1-4FD9-963A-2878E892053C">stdout stderr</rsp:DesiredStream>
    </rsp:Receive>
  </env:Body>
</env:Envelope>
, document out [EMPTY],
    at com.xebialabs.overthere.cifs.winrm.connector.ApacheHttpComponentsHttpClientHttpConnector.doSendMessage(ApacheHttpComponentsHttpClientHttpConnector.java:268)
    at com.xebialabs.overthere.cifs.winrm.connector.ApacheHttpComponentsHttpClientHttpConnector.sendMessage(ApacheHttpComponentsHttpClientHttpConnector.java:119)
    at com.xebialabs.overthere.cifs.winrm.WinRmClient.sendMessage(WinRmClient.java:210)
    at com.xebialabs.overthere.cifs.winrm.WinRmClient.receiveOutput(WinRmClient.java:122)
    at com.xebialabs.overthere.cifs.winrm.CifsWinRmConnection$1.run(CifsWinRmConnection.java:154)
Caused by: com.xebialabs.overthere.cifs.winrm.exception.WinRMRuntimeIOException: Response code was 500, document in [EMPTY], document out [EMPTY],10:57:42.373 [main] DEBUG c.x.overthere.cifs.winrm.WinRmClient - DeleteShell shellId 61B4EFC1-35B5-45DF-BB87-8CE4E757C54D

    at com.xebialabs.overthere.cifs.winrm.connector.ApacheHttpComponentsHttpClientHttpConnector.doSendMessage(ApacheHttpComponentsHttpClientHttpConnector.java:256)
    ... 4 more
10:57:42.374 [main] TRACE c.x.o.c.w.c.ApacheHttpComponentsHttpClientHttpConnector - Sending request to http://TEST-SEVEN:5985/wsman

[Vincent Partington]

Hi zerikv,

Well, I don't think developers that use the Overthere library would expect exceptions to be mixed with the stdout and stderr information that is sent to the OverthereProcessOutputHandler.

Then again, I do understand the use case. If the users of your application can only see the logs sent to the OverthereProcessOutputHandler, they would not see exceptions like these while you want to let hem know that the target host could not be reached.

We solved that in Deployit, the deployment automation product for which we built Overthere, by catching the exception and then invoking logError explicitly so that the error does show up in the logs in our user interface. For example:

try {
    connection.execute(processOutHandler, CmdLine.build("powershell.exe", "-nonInteractive", "-command", "Start-Sleep -s 12;"));
} catch(RuntimeIOException exc) {
    processOutHandler.logError("Could not execute Start-Sleep command", exc);
}

Hope this helps!

Regards, Vincent.

[zerikv]

Hi Vincent,

Thank you for your reply.

Unfortunately, catch the exception on the call connection#execute doesn't work for me.

My understanding of the issue is the following:

• The WinRMRuntimeIOException is thrown by ApacheHttpComponentsHttpClientHttpConnector#doSendMessage.
• This method is called from a thread started in CifsWinRmConnection#startProcess.
• This thread doesn't catch this kind of exception, so the exception is propagated
  and I guess finally catched by a UncaughtExceptionHandler which display it in the log.

Maybe the exception could be propagated by the OverthereProcess interface ?

What do you think about the following changes into CifsWinRmConnection#startProcess ?

1. Before the creation of thread processOutputReaderThread, add a wrapper for the exception:

final List<RuntimeIOException> exceptionWrapper = new ArrayList<RuntimeIOException>(1);

1. In the catch block of the run method in the thread processOutputReaderThread:

catch (Exception exc) {
    exceptionWrapper.add(new RuntimeIOException("Cannot start process " + commandLine, exc));
    return;
}

1. In the method OverthereProcess#waitFor, just before the return:

if(exceptionWrapper.size() != 0) {
    throw exceptionWrapper.get(0);
}

Thank you for your help,

Eric

[Vincent Partington]

Hi Eric,

Aha, like that! Yeah, I guess you'd want to have a way to get at that exception. But OverthereProcessOutputHandler doesn't seem to be the right place as that is only used for OverthereProcess.execute(), not for OverthereProcess.startProcess() which is what you are referring to.

Hmm, lemme have a think on this one...

Regards, Vincent.

[zerikv]

Hi Vincent,

Just a little precision: In fact I use the method OverthereConnection#execute(OverthereProcessOutputHandler, CmdLine) which is implemented in BaseOverthereConnection.

The method BaseOverthereConnection#execute calls the method startProcess(CmdLine) which is implemented for the protocol WinRM in CifsWinRmConnection#startProcess(CmdLine).

It is this last method which I was referring in my message above.

Thanks,

Eric.