Permissions Violation during subscribe unexpectedly closes the whole connection

Question

Permissions Violation during subscribe unexpectedly closes the whole connection

ArmorDarks opened this issue 2 years ago · comments

Client version: 2.8.0
Node version: 16.14.2

What's happening

If in subscription iterator for await (const message of subscription) happens Permissions Violation error like this

NatsError: 'Permissions Violation for Subscription to "subject"'
    at Function.toError (/code/node_modules/nats/lib/nats-base-client/protocol.js:306:20)
    at ProtocolHandler.<anonymous> (/code/node_modules/nats/lib/nats-base-client/protocol.js:339:41)
    at Generator.next (<anonymous>)
    at /code/node_modules/nats/lib/nats-base-client/protocol.js:8:71
    at new Promise (<anonymous>)
    at __awaiter (/code/node_modules/nats/lib/nats-base-client/protocol.js:4:12)
    at ProtocolHandler.processError (/code/node_modules/nats/lib/nats-base-client/protocol.js:337:16)
    at ProtocolHandler.push (/code/node_modules/nats/lib/nats-base-client/protocol.js:426:22)
    at Parser.parse (/code/node_modules/nats/lib/nats-base-client/parser.js:312:45)
    at ProtocolHandler.<anonymous> (/code/node_modules/nats/lib/nats-base-client/protocol.js:189:45)

it will close the whole NATS connection. Here's what connection.closed() will yield (same error):

NatsError: 'Permissions Violation for Subscription to "subject"'
    at Function.toError (/code/node_modules/nats/lib/nats-base-client/protocol.js:306:20)
    at ProtocolHandler.<anonymous> (/code/node_modules/nats/lib/nats-base-client/protocol.js:339:41)
    at Generator.next (<anonymous>)
    at /code/node_modules/nats/lib/nats-base-client/protocol.js:8:71
    at new Promise (<anonymous>)
    at __awaiter (/code/node_modules/nats/lib/nats-base-client/protocol.js:4:12)
    at ProtocolHandler.processError (/code/node_modules/nats/lib/nats-base-client/protocol.js:337:16)
    at ProtocolHandler.push (/code/node_modules/nats/lib/nats-base-client/protocol.js:426:22)
    at Parser.parse (/code/node_modules/nats/lib/nats-base-client/parser.js:312:45)
    at ProtocolHandler.<anonymous> (/code/node_modules/nats/lib/nats-base-client/protocol.js:189:45)

What's expected

I think closing the whole connection is very unexpected behavior here. While clearly, this is usually a human error when setting up NATS permissions, the fact that some subscriptions failed to doesn't mean that the whole connection should be killed because other subs are still able to work. It should be up to the application to decide what to do in such situations.

Alberto Ricart · Answer 1 · Tue Nov 15 2022 21:35:56 GMT+0800 (China Standard Time)

hmmm - that was the old behaviour for nats - but current behaviour changed (rather recently)
nats-io/nats.deno@afc6da4

https://github.com/nats-io/nats.deno/blob/main/tests/auth_test.ts#L130

Just did another paranoid type test, by adding this type of code to all those tests:

let err = await assertRejects(async () => {
    await Promise.race([nc.closed(), timeout(1000)])
  }, NatsError, "TIMEOUT");
  assertEquals(err.isPermissionError(), false);
  err = await assertRejects(async () => {
    await Promise.race([nc.closed(), timeout(1000)])
  }, NatsError, "TIMEOUT");
  assertEquals(err.isPermissionError(), false);
  assertEquals(nc.isClosed(), false);

And they pass.... you sure that the version of nats that you are running is what you think it is?

Alberto Ricart · Answer 2 · Tue Nov 15 2022 21:38:54 GMT+0800 (China Standard Time)

Ah - I think I see it - The client is closing for some reason - not an error, but the last error captured was the permissions error....

https://github.com/nats-io/nats.deno/blob/main/nats-base-client/protocol.ts#L465

Sergii Lavrin · Answer 3 · Tue Nov 15 2022 21:43:53 GMT+0800 (China Standard Time)

It might be, but it's always closing after an attempt to subscribe to the subject without permission. It's 100% reproducible.

Sorry, I've put the wrong version number. We're using 2.8.1-2 which was introduced for #524

Alberto Ricart · Answer 4 · Tue Nov 15 2022 22:02:09 GMT+0800 (China Standard Time)

@ArmorDarks can you copy your connection options exactly - I want to see all the reconnect options you are specifying.

Sergii Lavrin · Answer 5 · Tue Nov 15 2022 22:42:50 GMT+0800 (China Standard Time)

@aricart

const options = {
  maxReconnectAttempts: -1,
  name: 'service-name', // always static
  pass: 'pass', // always static
  servers: ['some-server-example.com:4333'], // always same URL here, leads to nats-operator behind the balancer
  user: 'user', // always static
  inboxPrefix: 'service-name.inbox', // always static
}

Alberto Ricart · Answer 6 · Tue Nov 15 2022 22:44:58 GMT+0800 (China Standard Time)

Also above you have two different permission errors for subject the other is for commerce.1757155.conversion

Alberto Ricart · Answer 7 · Tue Nov 15 2022 22:46:28 GMT+0800 (China Standard Time)

It might be, but it's always closing after an attempt to subscribe to the subject without permission. It's 100% reproducible.

Sorry, I've put the wrong version number. We're using 2.8.1-2 which was introduced for #524

Going to put the same test in node, perhaps something is not quite right there.

Sergii Lavrin · Answer 8 · Tue Nov 15 2022 22:48:31 GMT+0800 (China Standard Time)

Also above you have two different permission errors for subject the other is for "subject"

Forgot to edit the subject name in the log. Those were identical. Could you please remove the subject from your message too?

Alberto Ricart · Answer 9 · Wed Nov 16 2022 19:23:10 GMT+0800 (China Standard Time)

The reason for the close is not the permission error (unless the service is bailing the client).

Here's some code that you can run in your env to verify this:

mkdir /tmp/perm
cd /tmp/perm
npm init -y
npm install nats

Add the following server config as server.conf

port: 1234
authorization: {
    users: [
        {user: "a", password: "a", permissions: { subscribe: { deny: "q"}}}
    ]
}

And the following script:

import { connect, nuid } from "nats";

const nc = await connect({ port: 1234, user: "a", pass: "a" });

(async () => {
  for await (const s of nc.status()) {
    console.log(s);
  }
})();

let i = 0;
nc.subscribe("a.>", {
  callback: (err, msg) => {
    if (err) {
      console.error(`got error from sub to a.>: ${err.message}`);
      return;
    }
    i++;
    console.log(`got message: ${i}: ${msg.subject}`);
  },
});

setInterval(() => {
  nc.publish(`a.${nuid.next()}`);
}, 1000);

let subFails = 0;
setInterval(() => {
  nc.subscribe("q", {
    callback: (err, msg) => {
      subFails++;
      console.log(err.message);
    },
  });
}, 5000);

nc.closed().then((err) => {
  let m = "nats client closed";
  if (err) {
    m += `with error: ${err.message}`;
  }
  console.log(m);
});

You should see something like this - note that a new message is published every second, and the sub that would cause the permissions violation is attempted every 5 seconds:

node main.js
got message: 1: a.25XI8WQZ7X6QM0KD6MKLVV
got message: 2: a.25XI8WQZ7X6QM0KD6MKM0D
got message: 3: a.25XI8WQZ7X6QM0KD6MKM4V
got message: 4: a.25XI8WQZ7X6QM0KD6MKM9D
'Permissions Violation for Subscription to "q"'
{
  type: 'error',
  data: 'PERMISSIONS_VIOLATION',
  permissionContext: { operation: 'subscription', subject: 'q' }
}
got message: 5: a.25XI8WQZ7X6QM0KD6MKMDV
got message: 6: a.25XI8WQZ7X6QM0KD6MKMID
got message: 7: a.25XI8WQZ7X6QM0KD6MKMMV
got message: 8: a.25XI8WQZ7X6QM0KD6MKMRD
got message: 9: a.25XI8WQZ7X6QM0KD6MKMVV
'Permissions Violation for Subscription to "q"'
{
  type: 'error',
  data: 'PERMISSIONS_VIOLATION',
  permissionContext: { operation: 'subscription', subject: 'q' }
}
got message: 10: a.25XI8WQZ7X6QM0KD6MKN0D
got message: 11: a.25XI8WQZ7X6QM0KD6MKN4V
got message: 12: a.25XI8WQZ7X6QM0KD6MKN9D
got message: 13: a.25XI8WQZ7X6QM0KD6MKNDV
got message: 14: a.25XI8WQZ7X6QM0KD6MKNID
'Permissions Violation for Subscription to "q"'
{
  type: 'error',
  data: 'PERMISSIONS_VIOLATION',
  permissionContext: { operation: 'subscription', subject: 'q' }
}
got message: 15: a.25XI8WQZ7X6QM0KD6MKNMV
got message: 16: a.25XI8WQZ7X6QM0KD6MKNRD
got message: 17: a.25XI8WQZ7X6QM0KD6MKNVV
got message: 18: a.25XI8WQZ7X6QM0KD6MKO0D
got message: 19: a.25XI8WQZ7X6QM0KD6MKO4V
'Permissions Violation for Subscription to "q"'
{
  type: 'error',
  data: 'PERMISSIONS_VIOLATION',
  permissionContext: { operation: 'subscription', subject: 'q' }
}
got message: 20: a.25XI8WQZ7X6QM0KD6MKO9D
got message: 21: a.25XI8WQZ7X6QM0KD6MKODV
got message: 22: a.25XI8WQZ7X6QM0KD6MKOID
got message: 23: a.25XI8WQZ7X6QM0KD6MKOMV
got message: 24: a.25XI8WQZ7X6QM0KD6MKORD
'Permissions Violation for Subscription to "q"'
{
  type: 'error',
  data: 'PERMISSIONS_VIOLATION',
  permissionContext: { operation: 'subscription', subject: 'q' }
}
got message: 25: a.25XI8WQZ7X6QM0KD6MKOVV
got message: 26: a.25XI8WQZ7X6QM0KD6MKP0D
got message: 27: a.25XI8WQZ7X6QM0KD6MKP4V
got message: 28: a.25XI8WQZ7X6QM0KD6MKP9D
got message: 29: a.25XI8WQZ7X6QM0KD6MKPDV
'Permissions Violation for Subscription to "q"'
{
  type: 'error',
  data: 'PERMISSIONS_VIOLATION',
  permissionContext: { operation: 'subscription', subject: 'q' }
}
got message: 30: a.25XI8WQZ7X6QM0KD6MKPID
got message: 31: a.25XI8WQZ7X6QM0KD6MKPMV
got message: 32: a.25XI8WQZ7X6QM0KD6MKPRD
got message: 33: a.25XI8WQZ7X6QM0KD6MKPVV
got message: 34: a.25XI8WQZ7X6QM0KD6MKQ0D
'Permissions Violation for Subscription to "q"'
{
  type: 'error',
  data: 'PERMISSIONS_VIOLATION',
  permissionContext: { operation: 'subscription', subject: 'q' }
}
got message: 35: a.25XI8WQZ7X6QM0KD6MKQ4V
got message: 36: a.25XI8WQZ7X6QM0KD6MKQ9D
^C⏎

Alberto Ricart · Answer 10 · Wed Nov 16 2022 19:24:38 GMT+0800 (China Standard Time)

The closed error that you are seeing, I am not sure how it is triggering, I know where it copies, but the reason for the client bailing is still something we are not seeing. Are you still running the version that highlights the close error?

Alberto Ricart · Answer 11 · Wed Nov 16 2022 22:43:54 GMT+0800 (China Standard Time)

@ArmorDarks I am going to close this issue as well, because as shown in the example you can run above, a subscription error doesn't make the client close.

Sergii Lavrin · Answer 12 · Wed Nov 16 2022 23:44:27 GMT+0800 (China Standard Time)

Yeap, the issue is present only in v2.8.0.

With 2.8.1-2 I'm unable to reproduce it, so we're good to go 👍

Alberto Ricart · Answer 13 · Wed Nov 16 2022 23:50:48 GMT+0800 (China Standard Time)

I did see some different behaviour under 2.8.1-2 - for a couple of the tests, I would recommend we start from a known base with 2.9.0 and following the script for https://github.com/aricart/nats-close-tests

Sergii Lavrin · Answer 14 · Thu Nov 17 2022 01:19:05 GMT+0800 (China Standard Time)

@aricart Sure, I can deploy the 2.9.0 version tomorrow. Would you like to add any additional logging as you mentioned here nats-io/nats.deno#421?

Alberto Ricart · Answer 15 · Thu Nov 17 2022 02:02:38 GMT+0800 (China Standard Time)

I can do that for you

Sergii Lavrin · Answer 16 · Tue Nov 22 2022 02:25:22 GMT+0800 (China Standard Time)

@aricart Btw is it intended that the subscription won't be closed with an error in that case? Or it doesn't close to keep retrying?

Alberto Ricart · Answer 17 · Tue Nov 22 2022 02:47:37 GMT+0800 (China Standard Time)

The subscription won't restore. The server rejected it. You would have to resub. The client however will keep running and processing other subs

Sergii Lavrin · Answer 18 · Tue Nov 22 2022 02:49:20 GMT+0800 (China Standard Time)

@aricart That sound's logical. But then I guess the subscription should close with the error? subscription.closed never resolves or rejects with version 2.9.0.

Should I open a new issue about it?

Sergii Lavrin · Answer 19 · Tue Nov 22 2022 23:59:59 GMT+0800 (China Standard Time)

@aricart That seems to be the fix for this, right? nats-io/nats.deno@3b8e316

Alberto Ricart · Answer 20 · Thu Nov 24 2022 02:35:59 GMT+0800 (China Standard Time)

@aricart That sound's logical. But then I guess the subscription should close with the error? subscription.closed never resolves or rejects with version 2.9.0.

Should I open a new issue about it?

@ArmorDarks the subscription closes (not yet released - but in next for nats.js).The closed promises currently don't reject - they simply resolve when the subscription closes.

The resolving of the closed to an error (note not rejection because that would create a breaking condition for clients that don't expect a reject - and other closed type promises in nats don't reject is on my list to do.

2.9.0 doesn't contain the fix, nats.js@next does.

Sergii Lavrin · Answer 21 · Mon Nov 28 2022 20:40:56 GMT+0800 (China Standard Time)

@aricart I have a v2.9.1-0 on the server (which includes nats-io/nats.deno@3b8e316), and subscription still not reported as closed when that error happens.

Or nats-io/nats.deno@3b8e316 isn't a fix for that issue?

Alberto Ricart · Answer 22 · Mon Nov 28 2022 22:17:04 GMT+0800 (China Standard Time)

@ArmorDarks not sure what is going on this is what I did, and a sample program, and I see the subscription closed promise resolving, error reported by the callback, and error reported on the status. What node version are you using?

mkdir subclosed
cd subclosed
npm init -y
npm install nats@next

server.conf

debug: true,
trace: true,
authorization: {
    users: [
        {
            user: "a", password: "a", permissions: {subscribe: {deny: "q"}}
        }
    ]
}

main.js

import { connect } from "nats";

const nc = await connect({ user: "a", pass: "a", debug: true });

(async () => {
  for await (const s of nc.status()) {
    console.log(s);
  }
})();

const sub = nc.subscribe("q", {
  callback: (err, msg) => {
    if (err) {
      console.log(`callback: ${err.message}`);
    }
  },
});

sub.closed.then(() => {
    console.log(`subscription closed`);
})

 node main.js
> INFO {"server_id":"NCYFHXN4HY7JZZT3T7OO52KNHSHR6GIQKYRVTPYUB6CNZE3WV7XKTZSA","server_name":"NCYFHXN4HY7JZZT3T7OO52KNHSHR6GIQKYRVTPYUB6CNZE3WV7XKTZSA","version":"2.9.4","proto":1,"git_commit":"0b95294","go":"go1.19.2","host":"0.0.0.0","port":4222,"headers":true,"auth_required":true,"max_payload":1048576,"client_id":5,"client_ip":"127.0.0.1"} ␍␊
< CONNECT {"protocol":1,"version":"2.9.1-0","lang":"nats.js","verbose":false,"pedantic":false,"user":"a","pass":"a","headers":true,"no_responders":true}␍␊
< PING␍␊
> PONG␍␊
< SUB q 1␍␊
> -ERR 'Permissions Violation for Subscription to "q"'␍␊
callback: 'Permissions Violation for Subscription to "q"'
subscription closed
{
  type: 'error',
  data: 'PERMISSIONS_VIOLATION',
  permissionContext: { operation: 'subscription', subject: 'q' }
}
> PING␍␊
< PONG␍␊