Docker Swarm log issue caused by failure to remove an old container after docker stack deploy

[Ofir Germansky]

I have a stack for Confluent Kafka (including brokers, connectors, etc..).
I've restarted the stack (docker stack deploy) and run docker service logs on one of the services (cp-kafka_cp-kafka-connect1) and instead of getting the newest logs, I got the previous ones, meaning the ones that the previous container had (a container that the service task should have stopped and removed)

After the restart:

➤ docker service logs -f cp-kafka_cp-kafka-connect1
cp-kafka_cp-kafka-connect1.1.rjmam9btsc7l@cluster01 | Creating Secret File...
...
cp-kafka_cp-kafka-connect1.1.rjmam9btsc7l@cluster01 | [2021-12-02 16:51:12,072] INFO records [] (uk.co.org.task.Externalprojectconnector.ExternalProjectSourceTask)

I checked where the container for this single replica service got deployed, ssh to the VM and saw that Docker Swarm only stopped the old container, but it didn't remove it:

Notice the 'exited' cp-kafka_cp-kafka-connect1.1.rjmam9btsc7lwwxelrd81bz8w container which should have been removed:

➤ docker ps -a |grep connect1
afc2c62ad55a confluentinc/cp-kafka-connect:latest "/connect/setup_and_…" About an hour ago Up About an hour (healthy) 8083/tcp, 9092/tcp cp-kafka_cp-kafka-connect1.1.p2wuzvcns97nywkcl6h50nfer
911d6f22a2d3 confluentinc/cp-kafka-connect:latest "/connect/setup_and_…" 4 hours ago Exited (137) About an hour ago cp-kafka_cp-kafka-connect1.1.rjmam9btsc7lwwxelrd81bz8w

It is clear that instead of logs from cp-kafka_cp-kafka-connect1.1.p2wuzvcns97nywkcl6h50nfer I got logs from cp-kafka_cp-kafka-connect1.1.rjmam9btsc7lwwxelrd81bz8w

I then remove the 'exited' container:
➤ docker rm cp-kafka_cp-kafka-connect1.1.rjmam9btsc7lwwxelrd81bz8w cp-kafka_cp-kafka-connect1.1.rjmam9btsc7lwwxelrd81bz8w

And again run: docker service logs cp-kafka_cp-kafka-connect1
This time it was showing the latest logs as expected:

---

2021-12-02T16:52:04.443428257Z cp-kafka_cp-kafka-connect1.1.p2wuzvcns97n@cluster01 | Creating Secret File...
2021-12-02T16:52:04.446932866Z cp-kafka_cp-kafka-connect1.1.p2wuzvcns97n@cluster01 | Starting Kafka Connect...
...
2021-12-02T18:24:32.230561770Z cp-kafka_cp-kafka-connect1.1.p2wuzvcns97n@cluster01 | [2021-12-02 18:24:32,230] INFO records [] (uk.co.org.task.Externalprojectconnector.ExternalProjectSourceTask)

➤ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
mllkj8tx019ml4jhuelduhmju * cluster01 Ready Active Reachable 19.03.7
p6vdnwc7ahaogs09gfif70m2l cluster02 Ready Active Leader 19.03.7
vstjjddqusu2nw5rztt967mtq cluster03 Ready Active Reachable 19.03.7

➤ docker version
Client: Docker Engine - Community
Version: 19.03.6
API version: 1.40
Go version: go1.12.16
Git commit: 369ce74a3c
Built: Thu Feb 13 01:27:49 2020
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.7
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: 7141c199a2
Built: Wed Mar 4 01:21:08 2020
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683

➤ docker info
Client:
Debug Mode: false

Server:
Containers: 19
Running: 19
Paused: 0
Stopped: 0
Images: 23
Server Version: 19.03.7
Storage Driver: overlay2
Backing Filesystem:
Supports d_type: true
Native Overlay Diff: true
Logging Driver: local
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: mllkj8tx019ml4jhuelduhmju
Is Manager: true
ClusterID: pxrog2yr4vbj8tc0ndhf3j680
Managers: 3
Nodes: 5
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 10.12.1.215
Manager Addresses:
10.12.1.215:2377
10.12.1.216:2377
10.12.1.222:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-147-generic
Operating System: Ubuntu 18.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 23.54GiB
Name: cluster01
ID: JBN4:SSQD:K72A:USUE:3UPI:OG72:3IXQ:E7JJ:Q2RG:QATP:2RCQ:FEUW
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
nexus:9082
nexus:9083
127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support