Plugin fails when PostgreSQL connection is closed

[Gavin M. Roy]

Need to catch the process crashing for the pgsql connection which requires a refactor of how postgresql connections are managed.

[Bea Clune]

In testing, it looks like connection issues will result in a sock_closed error which crashes the pgsql_listen_worker. While the worker is correctly restarted by the supervisor, it appears that the PostgreSQL connection is not reset, since this only happens when the exchange is declared.

Furthermore, we can't call ensure_pgsql_connection from the worker upon restart (e.g. in init) for two reasons:

1. ensure_pgsql_connection is called per exchange (upon creation) and requires information from the exchange declaration to connect to PostgreSQL (hostname, port, etc.)
2. The closed PostgreSQL connection may not be purged from the dict that caches per-exchange connection information even after the socket is closed.

Is this a reasonable summary of the issue? Some thoughts:

1. Naively, it seems that pgsql_listen_lib:validate_pgsql_connection could attempt to recover from bad PostgreSQL state, but I'm not sure under what circumstances RabbitMQ attempts to validate exchanges so it may not solve the problem.
2. Handling this robustly seems challenging, and as you mentioned would likely require refactoring the PostgreSQL connection management logic.
3. Deleting and rebinding the exchange recovers from a bad connection without requiring a full RabbitMQ restart.