home-assistant / home-assistant-js-websocket

Getting an unhandled promise rejection when it's resubscribing to events after a reconnect and the socket drops the connection.

home-assistant-js-websocket/lib/connection.ts

Lines 156 to 159 in 5d14aee

    
           await this.sendMessagePromise( 
        
             messages.subscribeEvents(eventType), 
        
             commandId 
        
           );

Getting two different errors seems to be a race condition on which one gets thrown.

First One:

This is referencing the above SendMessagePromise

UnhandledPromiseRejectionWarning: Error: WebSocket is not open: readyState 2 (CLOSING)
    at WebSocket.send (/opt/node_modules/node-red-contrib-home-assistant-websocket/node_modules/ws/lib/websocket.js:314:19)
    at e.sendMessage (/opt/node_modules/home-assistant-js-websocket/dist/haws.umd.js:1:3458)
    at /opt/node_modules/home-assistant-js-websocket/dist/haws.umd.js:1:3630
    at new Promise (<anonymous>)
    at e.sendMessagePromise (/opt/node_modules/home-assistant-js-websocket/dist/haws.umd.js:1:3546)
    at p (/opt/node_modules/home-assistant-js-websocket/dist/haws.umd.js:1:8102)
    at w (/opt/node_modules/home-assistant-js-websocket/dist/haws.umd.js:1:8802)
    at /opt/node_modules/home-assistant-js-websocket/dist/haws.umd.js:1:7847
    at /opt/node_modules/home-assistant-js-websocket/dist/haws.umd.js:1:1440
    at Object.next (/opt/node_modules/home-assistant-js-websocket/dist/haws.umd.js:1:1545)

Second One:

home-assistant-js-websocket/lib/connection.ts

Lines 244 to 250 in 5d14aee

    
           // Reject in-flight requests 
        
           Object.keys(this.commands).forEach(id => { 
        
             const { reject } = this.commands[id]; 
        
             if (reject) { 
        
               reject(messages.error(ERR_CONNECTION_LOST, "Connection lost")); 
        
             } 
        
           });

Context:

Using node-red-contrib-home-assistant-websocket within hass.io and the addon-node-red. Everything works fine with the connection until you reboot hass.io and the addon is trying to reconnect. While trying to reconnect to Home Assistant it connects and disconnects from the hass.io proxy several times that's when the errors are getting thrown.

It's weird is that sendMessagePromise fails after a reconnect, because I would have expected it to fail already when trying to handle the auth phase (which is being done as part of createSocket.

Did you upgrade your ws dependency to the latest version so it properly handles the close event?

The latest ws update was not for the client side it was for the server side. But yes all the dependencies are up to date.

I mocked up a sample app, zachowj/hajw-test, that throws the second error. I can't seem to reproduce the first one with any consistency but only one ever happens not both at the same time.

[Client][Auth Phase] Initializing ws://127.0.0.1:8080
[Client][Auth Phase] New connection ws://127.0.0.1:8080
[Server] Sending need auth
[Client][Auth Phase] Received { type: 'auth_required', ha_version: '0.80.3' }
[Server] received: {"type":"auth","api_password":"abc"}
[Client][Auth Phase] Received { type: 'auth_ok', ha_version: '0.80.3' }
[Client] Sending SubscribeEntities
[Server] received: {"type":"subscribe_events","event_type":"state_changed","id":2}
[Server] Closing Connection to Client
[Server] received: {"type":"get_states","id":3}
[Client][UnhandledRejection] { type: 'result',
  success: false,
  error: { code: 3, message: 'Connection lost' } }
[Client][UnhandledRejection] { type: 'result',
  success: false,
  error: { code: 3, message: 'Connection lost' } }
[Client][Auth Phase] Initializing ws://127.0.0.1:8080
[Client][Auth Phase] New connection ws://127.0.0.1:8080
[Server] Sending need auth
[Client][Auth Phase] Received { type: 'auth_required', ha_version: '0.80.3' }
[Server] received: {"type":"auth","api_password":"abc"}
[Client][Auth Phase] Received { type: 'auth_ok', ha_version: '0.80.3' }
[Server] received: {"type":"get_states","id":2}

It happens when the client connects and authorizes with the server. Then sends SubscribeEntities to the server.

{"type":"subscribe_events","event_type":"state_changed","id":2}
{"type":"get_states","id":3}

At this point, the client is expecting the server to respond with a results message. If the server closes the connection before sending the result message the reject gets thrown in the socket close handler.

A possible fix would be wrapping the SendMessagePromise in a try/catch, not sure if that the correct fix.

Also another thing I noticed while mocking it up was that on the first connection the SubscribeEntities sends both

{"type":"subscribe_events","event_type":"state_changed","id":2}
{"type":"get_states","id":3}

to the server but after the rejection error any reconnect atempts after that the SubscribeEntites only sends {"type":"get_states","id":2} not actually both commands to the server.

Was able to reproduce the first error by having the server close the socket right after sending auth_ok message.

Working Example zachowj/hajw-test/tree/error1

Each error is dependent on when the connection gets closed from the server.

The subscribe to events should be automatically redone by the connection class:

home-assistant-js-websocket/lib/connection.ts

Lines 99 to 105 in 5d14aee

    
           if (info.eventCallback) { 
        
             this.subscribeEvents(info.eventCallback, info.eventType).then( 
        
               unsub => { 
        
                 info.unsubscribe = unsub; 
        
               } 
        
             ); 
        
           }

If that doesn't work, let's break it out into another issue. But let's first solve current one, as that might fix both.

About the second issue: I think that rejecting it with connection lost is, in absence of a proper retry mechanism, the appropriate thing to do. We tried to subscribe to entities and we failed. It would be up to the caller code to retry.

I think that a solution to sendMessagePromise raising an unknown error would be to reject it with the same error. We can check the readyState of the connection to know if we should raise or not. That's what ws does too: https://github.com/websockets/ws/blob/d2317b1d8dda7d36f1ba952f533ff45e2464ae52/lib/websocket.js#L313-L317

Aaah no, I see now that you're right.

We should not reject in-flight subscribeEvents commands in handleClose, because we are able to recover those.

hmm no, we should never reject a subscribeEvents in flight, because we don't store a reject in the command in flight.

Let me open a PR to add some types so it makes more sense.

aah okay, I think I know what's going on.

As part of subscribeEvents, we use sendMessagePromise, which is the one that is rejecting:

home-assistant-js-websocket/lib/connection.ts

Lines 156 to 159 in 5d14aee

    
           await this.sendMessagePromise( 
        
             messages.subscribeEvents(eventType), 
        
             commandId 
        
           );

And that makes it so that we don't retry either, as we never got to the point of storing our info in this.commands

PR to add some more types #61

PR to solve problem 2 up in #62

PR to solve problem 1 in #63

Did it solve it?

It has fixed the first issue and it does resubscription to the correct events on a reconnect but I am still seeing an unhandled rejection when using .subscribeEntities or .subscribeEvents when the server disconnects before sending back a result.

Sorry, haven't had a chance to look into it further.

Not sure if I should have opened a new issue for this but here goes.

Since subscribeEvents returns info.unsubscribe and not the promise there is no way to catch a rejection from the promise inside. This rejection occurs when the promise in subscribeEvents is called and has not been resolved by the time the connection is closed. The _handleClose is where the reject is thrown.

home-assistant-js-websocket/lib/connection.ts

Lines 169 to 200 in 2bc4cac

    
           async subscribeEvents<EventType>( 
        
             eventCallback: (ev: EventType) => void, 
        
             eventType?: string 
        
           ) { 
        
             // Command ID that will be used 
        
             const commandId = this._genCmdId(); 
        
             let info: SubscribeEventCommmandInFlight; 
        
             await new Promise((resolve, reject) => { 
        
               // We store unsubscribe on info object. That way we can overwrite it in case 
        
               // we get disconnected and we have to subscribe again. 
        
               info = this.commands[commandId] = { 
        
                 resolve, 
        
                 reject, 
        
                 eventCallback: eventCallback as (ev: any) => void, 
        
                 eventType, 
        
                 unsubscribe: async () => { 
        
                   await this.sendMessagePromise(messages.unsubscribeEvents(commandId)); 
        
                   delete this.commands[commandId]; 
        
                 } 
        
               }; 
        
               try { 
        
                 this.sendMessage(messages.subscribeEvents(eventType), commandId); 
        
               } catch (err) { 
        
                 // Happens when the websocket is already closing. 
        
                 // Don't have to handle the error, reconnect logic will pick it up. 
        
               } 
        
             }); 
        
             return () => info.unsubscribe(); 
        
           }

A simple solution is to just use a .catch(() => {}) on the promise. The same reasoning can be used here as you did for the SendMessage try/catch block in the promise

// Happens when the websocket is already closing.
// Don't have to handle the error, reconnect logic will pick it up.

Extracted your comment into a new issue.

	await this.sendMessagePromise(
	messages.subscribeEvents(eventType),
	commandId
	);

	// Reject in-flight requests
	Object.keys(this.commands).forEach(id => {
	const { reject } = this.commands[id];
	if (reject) {
	reject(messages.error(ERR_CONNECTION_LOST, "Connection lost"));
	}
	});

	if (info.eventCallback) {
	this.subscribeEvents(info.eventCallback, info.eventType).then(
	unsub => {
	info.unsubscribe = unsub;
	}
	);
	}

	async subscribeEvents<EventType>(
	eventCallback: (ev: EventType) => void,
	eventType?: string
	) {
	// Command ID that will be used
	const commandId = this._genCmdId();
	let info: SubscribeEventCommmandInFlight;

	await new Promise((resolve, reject) => {
	// We store unsubscribe on info object. That way we can overwrite it in case
	// we get disconnected and we have to subscribe again.
	info = this.commands[commandId] = {
	resolve,
	reject,
	eventCallback: eventCallback as (ev: any) => void,
	eventType,
	unsubscribe: async () => {
	await this.sendMessagePromise(messages.unsubscribeEvents(commandId));
	delete this.commands[commandId];
	}
	};

	try {
	this.sendMessage(messages.subscribeEvents(eventType), commandId);
	} catch (err) {
	// Happens when the websocket is already closing.
	// Don't have to handle the error, reconnect logic will pick it up.
	}
	});

	return () => info.unsubscribe();
	}

Unhandled Promise Rejection after a reconnection and trying to resubscribe to events