long node name seems not be ok on k8s which set by using RABBITMQ_USE_LONGNAME
kaybinwong opened this issue · comments
long node name using RABBITMQ_USE_LONGNAME=true seems not be ok on k8s, what had i done wrongly? @gmr @xek
logs and yaml file are below~~
thanks~
log below
[root@node-138 rabbitmq-autocluster]# kubectl logs -f po/rabbitmq-789413866-v8znm
RabbitMQ 3.6.6. Copyright (C) 2007-2016 Pivotal Software, Inc.
## ## Licensed under the MPL. See http://www.rabbitmq.com/
## ##
########## Logs: tty
###### ## tty
##########
Starting broker...
=INFO REPORT==== 21-Mar-2017::13:45:45 ===
Starting RabbitMQ 3.6.6 on Erlang 18.3.2
Copyright (C) 2007-2016 Pivotal Software, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/
=INFO REPORT==== 21-Mar-2017::13:45:45 ===
node : rabbit@rabbitmq-789413866-v8znm
home dir : /var/lib/rabbitmq
config file(s) : /usr/lib/rabbitmq/etc/rabbitmq/rabbitmq.config
cookie hash : iqG7DCBA+lxNNLQq/Y6efg==
log : tty
sasl log : tty
database dir : /var/lib/rabbitmq/mnesia
=INFO REPORT==== 21-Mar-2017::13:45:50 ===
autocluster: log level set to debug
=INFO REPORT==== 21-Mar-2017::13:45:50 ===
autocluster: Running discover/join stage
=INFO REPORT==== 21-Mar-2017::13:45:50 ===
application: mnesia
exited: stopped
type: temporary
=INFO REPORT==== 21-Mar-2017::13:45:50 ===
autocluster: Apps 'rabbit' and 'mnesia' successfully stopped
=INFO REPORT==== 21-Mar-2017::13:45:50 ===
autocluster: Running step initialize_backend
=INFO REPORT==== 21-Mar-2017::13:45:50 ===
autocluster: Using etcd backend
=INFO REPORT==== 21-Mar-2017::13:45:50 ===
autocluster: Running step acquire_startup_lock
=INFO REPORT==== 21-Mar-2017::13:45:50 ===
autocluster: PUT https://etcd-lb:2379/v2/keys/rabbitmq/default/startup_lock?prevExist=false&ttl=30 ["value=rabbit%40rabbitmq-789413866-v8znm%20-%20ysmxpeomyamfmsiilycuxschnlaxgiqm"]
=INFO REPORT==== 21-Mar-2017::13:45:50 ===
autocluster: Response: [{ok,{{"HTTP/1.1",201,"Created"},
[{"date","Tue, 21 Mar 2017 13:45:50 GMT"},
{"content-length","245"},
{"content-type","application/json"},
{"x-etcd-cluster-id","d4ae3948b2497eef"},
{"x-etcd-index","699123"},
{"x-raft-index","3656297"},
{"x-raft-term","2178"}],
"{\"action\":\"create\",\"node\":{\"key\":\"/rabbitmq/default/startup_lock\",\"value\":\"rabbit@rabbitmq-789413866-v8znm - ysmxpeomyamfmsiilycuxschnlaxgiqm\",\"expiration\":\"2017-03-21T13:46:20.631031784Z\",\"ttl\":30,\"modifiedIndex\":699123,\"createdIndex\":699123}}\n"}}]
=INFO REPORT==== 21-Mar-2017::13:45:50 ===
autocluster: Starting startup lock refresher
=INFO REPORT==== 21-Mar-2017::13:45:50 ===
autocluster: Startup lock acquired
=INFO REPORT==== 21-Mar-2017::13:45:50 ===
autocluster: Running step find_best_node_to_join
=INFO REPORT==== 21-Mar-2017::13:45:50 ===
autocluster: GET https://etcd-lb:2379/v2/keys/rabbitmq/default/nodes?recursive=true
=INFO REPORT==== 21-Mar-2017::13:45:50 ===
autocluster: Response: [{ok,{{"HTTP/1.1",200,"OK"},
[{"date","Tue, 21 Mar 2017 13:45:50 GMT"},
{"content-length","492"},
{"content-type","application/json"},
{"x-etcd-cluster-id","d4ae3948b2497eef"},
{"x-etcd-index","699123"},
{"x-raft-index","3656297"},
{"x-raft-term","2178"}],
"{\"action\":\"get\",\"node\":{\"key\":\"/rabbitmq/default/nodes\",\"dir\":true,\"nodes\":[{\"key\":\"/rabbitmq/default/nodes/rabbit@rabbitmq-789413866-rdfdf\",\"value\":\"enabled\",\"expiration\":\"2017-03-21T13:46:17.760041391Z\",\"ttl\":28,\"modifiedIndex\":699116,\"createdIndex\":699116},{\"key\":\"/rabbitmq/default/nodes/rabbit@rabbitmq-789413866-602f4\",\"value\":\"enabled\",\"expiration\":\"2017-03-21T13:46:18.968415025Z\",\"ttl\":29,\"modifiedIndex\":699119,\"createdIndex\":699119}],\"modifiedIndex\":668913,\"createdIndex\":668913}}\n"}}]
=INFO REPORT==== 21-Mar-2017::13:45:50 ===
autocluster: List of nodes from backend: ['rabbit@rabbitmq-789413866-rdfdf',
'rabbit@rabbitmq-789413866-602f4']
=INFO REPORT==== 21-Mar-2017::13:45:51 ===
autocluster: Best discovery node choice: undefined
=INFO REPORT==== 21-Mar-2017::13:45:51 ===
autocluster: Running step maybe_cluster
=INFO REPORT==== 21-Mar-2017::13:45:51 ===
autocluster: We are the first node in the cluster, starting up unconditionally.
=INFO REPORT==== 21-Mar-2017::13:45:51 ===
autocluster: Starting back 'rabbit' application
=INFO REPORT==== 21-Mar-2017::13:45:51 ===
Memory limit set to 2265MB of 2831MB total.
=INFO REPORT==== 21-Mar-2017::13:45:51 ===
Disk free limit set to 50MB
=INFO REPORT==== 21-Mar-2017::13:45:51 ===
Limiting to approx 1048476 file handles (943626 sockets)
=INFO REPORT==== 21-Mar-2017::13:45:51 ===
FHC read buffering: OFF
FHC write buffering: OFF
=INFO REPORT==== 21-Mar-2017::13:45:51 ===
Database directory at /var/lib/rabbitmq/mnesia is empty. Initialising from scratch...
=INFO REPORT==== 21-Mar-2017::13:45:51 ===
application: mnesia
exited: stopped
type: temporary
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Priority queues enabled, real BQ is rabbit_variable_queue
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Starting rabbit_node_monitor
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Management plugin: using rates mode 'basic'
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide index
=WARNING REPORT==== 21-Mar-2017::13:45:52 ===
msg_store_persistent: rebuilding indices from scratch
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Adding vhost '/'
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Creating user 'guest'
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Setting user tags for user 'guest' to [administrator]
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Setting permissions for 'guest' in '/' to '.*', '.*', '.*'
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
started TCP Listener on [::]:5672
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
autocluster: Running register stage
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
autocluster: Running step register_in_backend
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
autocluster: PUT https://etcd-lb:2379/v2/keys/rabbitmq/default/nodes/rabbit%40rabbitmq-789413866-v8znm?ttl=30 ["value=enabled"]
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
autocluster: Response: [{ok,{{"HTTP/1.1",201,"Created"},
[{"date","Tue, 21 Mar 2017 13:45:52 GMT"},
{"content-length","208"},
{"content-type","application/json"},
{"x-etcd-cluster-id","d4ae3948b2497eef"},
{"x-etcd-index","699126"},
{"x-raft-index","3656313"},
{"x-raft-term","2178"}],
"{\"action\":\"set\",\"node\":{\"key\":\"/rabbitmq/default/nodes/rabbit@rabbitmq-789413866-v8znm\",\"value\":\"enabled\",\"expiration\":\"2017-03-21T13:46:22.032313738Z\",\"ttl\":30,\"modifiedIndex\":699126,\"createdIndex\":699126}}\n"}}]
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
autocluster: Updated node registration with etcd
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
autocluster: Starting etcd TTL node key updater
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
autocluster: Running unlock stage
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
autocluster: Running step release_startup_lock
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
autocluster: Stopped startup lock refresher
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
autocluster: DELETE https://etcd-lb:2379/v2/keys/rabbitmq/default/startup_lock?prevExist=true&prevValue=rabbit%40rabbitmq-789413866-v8znm%20-%20ysmxpeomyamfmsiilycuxschnlaxgiqm [[]]
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
autocluster: Response: [{ok,{{"HTTP/1.1",200,"OK"},
[{"date","Tue, 21 Mar 2017 13:45:52 GMT"},
{"content-length","352"},
{"content-type","application/json"},
{"x-etcd-cluster-id","d4ae3948b2497eef"},
{"x-etcd-index","699127"},
{"x-raft-index","3656314"},
{"x-raft-term","2178"}],
"{\"action\":\"compareAndDelete\",\"node\":{\"key\":\"/rabbitmq/default/startup_lock\",\"modifiedIndex\":699127,\"createdIndex\":699123},\"prevNode\":{\"key\":\"/rabbitmq/default/startup_lock\",\"value\":\"rabbit@rabbitmq-789413866-v8znm - ysmxpeomyamfmsiilycuxschnlaxgiqm\",\"expiration\":\"2017-03-21T13:46:20.631031784Z\",\"ttl\":29,\"modifiedIndex\":699123,\"createdIndex\":699123}}\n"}}]
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
MQTT retained message store: rabbit_mqtt_retained_msg_store_dets
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
started MQTT TCP Listener on [::]:1883
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
rabbit_stomp: default user 'guest' enabled
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
started STOMP TCP Listener on [::]:61613
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
rabbit_web_stomp: listening for HTTP connections on 0.0.0.0:15674
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Management plugin started. Port: 15672
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics event collector started.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics channel stats collector started.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics queue stats collector started.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics database started.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_queue_stats_fine_stats with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_queue_stats_deliver_get with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_queue_stats_queue_msg_counts with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_queue_stats_queue_msg_rates with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_queue_stats_process_stats with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_queue_exchange_stats_fine_stats with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_vhost_stats_deliver_get with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_vhost_stats_fine_stats with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_vhost_stats_queue_msg_rates with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_vhost_stats_queue_msg_counts with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_vhost_stats_coarse_conn_stats with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_channel_queue_stats_deliver_get with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_channel_queue_stats_fine_stats with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_channel_queue_stats_queue_msg_counts with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_channel_stats_deliver_get with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_channel_stats_fine_stats with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_channel_stats_queue_msg_counts with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_channel_stats_process_stats with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_channel_exchange_stats_deliver_get with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_channel_exchange_stats_fine_stats with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_exchange_stats_fine_stats with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_node_stats_coarse_node_stats with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_node_node_stats_coarse_node_node_stats with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_connection_stats_coarse_conn_stats with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table aggr_connection_stats_process_stats with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table channel_stats with interval 5000.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Statistics garbage collector started for table connection_stats with interval 5000.
completed with 18 plugins.
=INFO REPORT==== 21-Mar-2017::13:45:52 ===
Server startup complete; 18 plugins started.
* rabbitmq_shovel_management
* rabbitmq_federation_management
* rabbitmq_management
* rabbitmq_web_stomp
* cowboy
* cowlib
* rabbitmq_web_dispatch
* webmachine
* mochiweb
* rabbitmq_management_agent
* rabbitmq_stomp
* rabbitmq_consistent_hash_exchange
* autocluster
* rabbitmq_federation
* rabbitmq_shovel
* rabbitmq_mqtt
* amqp_client
* sockjs
=INFO REPORT==== 21-Mar-2017::13:46:07 ===
autocluster: PUT https://etcd-lb:2379/v2/keys/rabbitmq/default/nodes/rabbit%40rabbitmq-789413866-v8znm?ttl=30 ["value=enabled"]
=INFO REPORT==== 21-Mar-2017::13:46:07 ===
autocluster: Response: [{ok,{{"HTTP/1.1",200,"OK"},
[{"date","Tue, 21 Mar 2017 13:46:07 GMT"},
{"content-length","403"},
{"content-type","application/json"},
{"x-etcd-cluster-id","d4ae3948b2497eef"},
{"x-etcd-index","699155"},
{"x-raft-index","3656452"},
{"x-raft-term","2178"}],
"{\"action\":\"set\",\"node\":{\"key\":\"/rabbitmq/default/nodes/rabbit@rabbitmq-789413866-v8znm\",\"value\":\"enabled\",\"expiration\":\"2017-03-21T13:46:37.040084768Z\",\"ttl\":30,\"modifiedIndex\":699155,\"createdIndex\":699155},\"prevNode\":{\"key\":\"/rabbitmq/default/nodes/rabbit@rabbitmq-789413866-v8znm\",\"value\":\"enabled\",\"expiration\":\"2017-03-21T13:46:22.032313738Z\",\"ttl\":15,\"modifiedIndex\":699126,\"createdIndex\":699126}}\n"}}]
=INFO REPORT==== 21-Mar-2017::13:46:07 ===
autocluster: Updated node registration with etcd
=INFO REPORT==== 21-Mar-2017::13:46:22 ===
autocluster: PUT https://etcd-lb:2379/v2/keys/rabbitmq/default/nodes/rabbit%40rabbitmq-789413866-v8znm?ttl=30 ["value=enabled"]
=INFO REPORT==== 21-Mar-2017::13:46:22 ===
autocluster: Response: [{ok,{{"HTTP/1.1",200,"OK"},
[{"date","Tue, 21 Mar 2017 13:46:22 GMT"},
{"content-length","403"},
{"content-type","application/json"},
{"x-etcd-cluster-id","d4ae3948b2497eef"},
{"x-etcd-index","699182"},
{"x-raft-index","3656595"},
{"x-raft-term","2178"}],
"{\"action\":\"set\",\"node\":{\"key\":\"/rabbitmq/default/nodes/rabbit@rabbitmq-789413866-v8znm\",\"value\":\"enabled\",\"expiration\":\"2017-03-21T13:46:52.043102962Z\",\"ttl\":30,\"modifiedIndex\":699182,\"createdIndex\":699182},\"prevNode\":{\"key\":\"/rabbitmq/default/nodes/rabbit@rabbitmq-789413866-v8znm\",\"value\":\"enabled\",\"expiration\":\"2017-03-21T13:46:37.040084768Z\",\"ttl\":15,\"modifiedIndex\":699155,\"createdIndex\":699155}}\n"}}]
=INFO REPORT==== 21-Mar-2017::13:46:22 ===
autocluster: Updated node registration with etcd
rabbitmq.yaml
# see at https://github.com/kaybinwong/rabbitmq-autocluster
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: rabbitmq
labels:
app: rabbitmq
spec:
replicas: 3
template:
metadata:
labels:
app: rabbitmq
spec:
containers:
- name: rabbitmq-cluster
image: 192.168.1.138:5000/poker/rabbitmq-autocluster
imagePullPolicy: Always
ports:
- containerPort: 5671
name: queue-ssl
- containerPort: 5672
name: queue-rw-port
- containerPort: 15672
name: queue-mgt-port
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: AUTOCLUSTER_LOG_LEVEL
value: debug
- name: AUTOCLUSTER_TYPE
value: etcd
- name: ETCD_SCHEME
value: https
- name: ETCD_HOST
value: "etcd-lb"
- name: ETCD_PORT
value: "2379"
- name: ETCD_PREFIX
value: rabbitmq
- name: ETCD_TTL
value: "15"
- name: RABBITMQ_USE_LONGNAME
value: "true"
imagePullSecrets:
- name: hub.poker.io.key
---
kind: Service
apiVersion: v1
metadata:
labels:
app: rabbitmq
name: rabbitmq
spec:
ports:
- port: 5671
targetPort: 5671
name: queue-ssl
- port: 5672
targetPort: 5672
name: queue-rw-port
- port: 15672
targetPort: 15672
name: queue-mgt-port
selector:
app: rabbitmq
I'm sorry but unless you provide more details than "is not OK" we cannot help you.
According to the log all operations succeed. So what's exactly the issue?
it is my fault~ @michaelklishin
the logs seems nothing wrong, but when i check the cluster status by cli below, it gets error.
the field USE_LONGNAME which set by RABBITMQ_USE_LONGNAME does not work.
cli
[root@node-138 rabbitmq-autocluster]# kubectl exec rabbitmq-3544912728-hxdrb rabbitmqctl cluster_status
Cluster status of node 'rabbit@rabbitmq-3544912728-hxdrb' ...
Error: unable to connect to node 'rabbit@rabbitmq-3544912728-hxdrb': nodedown
DIAGNOSTICS
===========
attempted to contact: ['rabbit@rabbitmq-3544912728-hxdrb']
rabbit@rabbitmq-3544912728-hxdrb:
* connected to epmd (port 4369) on rabbitmq-3544912728-hxdrb
* epmd reports node 'rabbit' running on port 25672
* TCP connection succeeded but Erlang distribution failed
* suggestion: hostname mismatch?
* suggestion: is the cookie set correctly?
* suggestion: is the Erlang distribution using TLS?
current node details:
- node name: 'rabbitmq-cli-74@rabbitmq-3544912728-hxdrb.no-domain'
- home dir: /var/lib/rabbitmq
- cookie hash: iqG7DCBA+lxNNLQq/Y6efg==
@kaybinwong well, there's a hint right in the message:
* connected to epmd (port 4369) on rabbitmq-3544912728-hxdrb
* epmd reports node 'rabbit' running on port 25672
* TCP connection succeeded but Erlang distribution failed
* suggestion: hostname mismatch?
* suggestion: is the cookie set correctly?
* suggestion: is the Erlang distribution using TLS?
As the README for this plugin suggests, it is not a replacement for understanding of how RabbitMQ nodes and CLI tools communicate and authenticate. See How Nodes and CLI Tools Authenticate in the clustering guide. There is no shortage of similar questions on rabbitmq-users pretty much every week.