m3db / m3

M3 monorepo - Distributed TSDB, Aggregator and Query Engine, Prometheus Sidecar, Graphite Compatible, Metrics Platform

Home Page:https://m3db.io/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

can't fix up cluster when add a wrong placement

angelala00 opened this issue · comments

commented

Currently, if user added a node to placement, then cluster is hard to fix up. I can not do anything to fix this condition if just by m3db api.

For example,

add a broken node
curl -X POST localhost:7201/api/v1/services/m3db/placement -d '{
"instances": [
{
"id": "node-4",
"isolationGroup": "a",
"zone": "embedded",
"weight": 100,
"endpoint": "x.x.x.x",
"hostname": "broken-node",
"port": 9000
}
]
}'
than i find endpoint without port ,and wanna remove this node, everything is stuck.
{"error":"instances [node-1,node-2,node-3,node-4] do not have all shards available"}
How can i fix this?

commented

same question

commented

fixed