Giters

open-contracting / deploy

Deployment configuration and scripts

Home Page:https://ocdsdeploy.readthedocs.io/en/latest/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Make Docker stop before RabbitMQ, if possible

jpmckinney opened this issue · comments

commented

When a server is restarted, RabbitMQ stops and then Docker stops. When RabbitMQ becomes unavailable, the Docker containers end up reporting a bunch of errors to Sentry. If we shut down Docker first, they won't do that.

I don't know if perhaps there's something we can do in /etc/init.d.

Applies to ocp13 and ocp23.

From testing locally:

  1. Ensure RabbitMQ is running

  2. Run kingfisher-process ./manage.py runserver in a tab

  3. Start a crawl:

     env RABBIT_URL=amqp://127.0.0.1 KINGFISHER_API2_URL=http://localhost:8000 RABBIT_EXCHANGE_NAME=kingfisher_process_test RABBIT_ROUTING_KEY=kingfisher_process_test_api scrapy crawl delay -s SPIDER_MODULES=tests.spiders

  4. Stop RabbitMQ

Produces crawl log like:

2024-02-16 14:12:05 [yapw.clients] WARNING: Channel 1 was closed: ConnectionClosedByBroker: (320) "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2024-02-16 14:12:05 [yapw.clients] WARNING: Connection closed, reconnecting in 15s: ConnectionClosedByBroker: (320) "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"

...

2024-02-16 14:12:20 [pika.adapters.utils.io_services_utils] ERROR: Socket failed to connect: <socket.socket fd=14, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 54970)>; error=61 (Connection refused)
2024-02-16 14:12:20 [pika.adapters.utils.connection_workflow] ERROR: TCP Connection attempt failed: ConnectionRefusedError(61, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 5672))
2024-02-16 14:12:20 [pika.adapters.utils.connection_workflow] ERROR: AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(61, 'Connection refused')
2024-02-16 14:12:20 [pika.adapters.utils.connection_workflow] ERROR: AMQP connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorSocketConnectError: ConnectionRefusedError(61, 'Connection refused'); first exception - None.
2024-02-16 14:12:20 [pika.adapters.utils.connection_workflow] ERROR: AMQPConnectionWorkflow - reporting failure: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorSocketConnectError: ConnectionRefusedError(61, 'Connection refused'); first exception - None
2024-02-16 14:12:20 [pika.adapters.base_connection] ERROR: Full-stack connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorSocketConnectError: ConnectionRefusedError(61, 'Connection refused'); first exception - None
2024-02-16 14:12:20 [pika.adapters.base_connection] ERROR: Self-initiated stack bring-up failed: AMQPConnectionError: (AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorSocketConnectError: ConnectionRefusedError(61, 'Connection refused'); first exception - None,)
2024-02-16 14:12:20 [yapw.clients] ERROR: Connection failed, retrying in 15s: AMQPConnectionError: (AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorSocketConnectError: ConnectionRefusedError(61, 'Connection refused'); first exception - None,)