Possible port lock situation when trying to connect to a local, not running, NATS server
LaurensVergote opened this issue · comments
Observed behavior
When creating a connection to a local hostname, it's possible that the TcpClient created will grab the target port as its local port. This seems to result in a connection successfully being established and the client application grabbing the port, making it impossible for the nats server to ever start up later on using that port.
output of netstat (table format for readability):
netstat -abon | findstr 4222
| Proto | Local Address | Foreign Address | State | PID |
|---|---|---|---|---|
| TCP | 10.3.1.54:4222 | 10.3.1.54:4222 | ESTABLISHED | 4708 |
Possible solutions:
- Use the TcpClient constructor with IPEndpoint input (TcpClient)
- After constructing the TcpClient, modify the internal socket (TcpClient.Client) to exclude the target port from being used as the local endpoint's port
Comments:
While I'm not 100% certain on the root cause of this, the above description is my best educated guess on why this would happen.
I've only noticed it on dev machines that run the host process, which tries to connect every 1 second to NATS, and while the target local nats-server, is not running.
This results in not being able to start the local nats-server anymore.
I know it's very niche but it's better if its noted down at least, even if there's no intent to fix it or spend time on it.
Expected behavior
The NATS client does not use the target endpoint's port as the local endpoint's when connecting with a local endpoint
Server and client version
Server version not relevant.
client version: 1.1.1
Host environment
Running on Windows 10 Enterprise, loading the NATS client library as .netstandard 2.0 from a .NET 6.0 host process.
Steps to reproduce
Continually call Connect until you randomly get the target's endpoint port as your local endpoint's port.
I don't see it. Why would Connection.cs#L386 touch any port at all, and why would the ensuing code touch any port other than the specified port.
If it does, and it's inside the TcpClient code, this would affect every C# project ever.
Do you have any local name resolution in the environment?
Are you possibly starting a second server somehow?
Can you post your "Continually call Connect" code?
Hi @scottf, I created a small console app that simulates this problem
https://gist.github.com/LaurensVergote/f4270f3a22ea4960eb4dfadc2ed63dc9
It's not pretty code since it's cobbled together, but should demonstrate what I mean is happening.
You can put a breakpoint on the Console.WriteLine to inspect the objects.
In normal circumstances, the ConnectAsync() call will throw an exception saying there is nothing running on the target port.
After some further investigation, I noticed that this issue does not happen on an initial connect, only on a reconnect.
I currently have a fix that works in a test application, will put it in the NATS.Client code to test if it resolves the issue.
Reproduction using just the NATS.Client library: https://gist.github.com/LaurensVergote/ba4b4ea1825f2ff5fed3d1aee9214dd8
Just make sure a nats-server.exe is available next to the compiled binary