Error when connecting to Cassandra cluster

[rhit-liuy29]

Hi,

Thank you so much for the library. I'm new to Elixir and Phoenix, and I'm using Xandra in my school project. Currently, I have a connection problem with Cassandra in cluster mode.

The error I'm getting

[error] Process #PID<0.568.0> raised an exception
** (Xandra.ConnectionError) action "checkout from cluster :myXandra" failed with reason: not connected to any of the nodes       
    (xandra 0.18.1) lib/xandra/cluster.ex:408: Xandra.Cluster.prepare!/3
    (socket_test 0.1.0) lib/socket_test_web/channels/clump_message.ex:60: HelloWeb.ClumpMessage.save_message/8

Sometimes, I can connect to Cassandra fine, but a lot of the time I get the above error.

What I tried

• I looked through the documentation and experimented with :sync_connect, timeout for queue_checkouts_before_connecting
• I can connect and perform queries fine in normal Xandra (without cluster).
• Other online search

But the problem still persists.

My project setup

Phoenix:

config.ex

config :socket_test, :xandra,
  nodes: ["localhost:9042", "localhost:9043", "localhost:9044"],
  authentication: {Xandra.Authenticator.Password,
        [
          username: "username",
          password: "password",
        ]
  },
  keyspace: "project",
  name: :myXandra,
  sync_connect: :infinity

application.ex

child = [
...
{Xandra.Cluster, Application.fetch_env!(:socket_test, :xandra)},
...
]

clump_message.ex

  defp save_message(some_params) do
    prepared = Xandra.Cluster.prepare!(:myXandra,
     "INSERT INTO chat_message_by_clump (some_fields)
     VALUES (:some_fields)")


    data = %{
      some data
    }

     Xandra.Cluster.execute!(:myXandra, prepared, data)
  end

Cassandra:

3 nodes setup with docker
docker-compose.yaml

version: '3.7'

services:
  cass1:
    image: cassandra:4.1.3 # better to use a specific version, if you want to control upgrades 
    container_name: cass1
    hostname: cass1
    healthcheck:
      test: [ "CMD-SHELL", "nodetool status" ]
      interval: 5s
      timeout: 10s
      retries: 60
    ports:
      - "9042:9042" # Expose native binary CQL port for your apps
    volumes:
      - ../../../DB/cass1:/var/lib/cassandra    # This is the volume that will persist data for cass1 node
      - ../../../DB/cass3-config:/etc/cassandra   # Use your own config files for full control
    environment: &environment    # Declare and save environments variables into "environment"
      CASSANDRA_SEEDS: "cass1, cass2"    # The first two nodes will be seeds
      CASSANDRA_CLUSTER_NAME: my_cluster
      CASSANDRA_DC: IND_TH
      CASSANDRA_RACK: Rack0
      CASSANDRA_ENDPOINT_SNITCH: GossipingPropertyFileSnitch
      CASSANDRA_NUM_TOKENS: 128

  cass2:
    image: cassandra:4.1.3
    container_name: cass2
    hostname: cass2
    healthcheck:
      test: [ "CMD-SHELL", "nodetool status" ]
      interval: 5s
      timeout: 10s
      retries: 60
    ports:
      - "9043:9042" # Expose native binary CQL port for your apps
    volumes:
      - ../../../DB/cass2:/var/lib/cassandra    # This is the volume that will persist data for cass2 node
      - ../../../DB/cass3-config:/etc/cassandra   # Use your own config files for full contro
    environment: *environment
    depends_on:
      cass1:
        # start cass2 only after cass1 is healthy
        condition: service_healthy

  cass3:
    image: cassandra:4.1.3
    container_name: cass3
    hostname: cass3
    healthcheck:
      test: [ "CMD-SHELL", "nodetool status" ]
      interval: 5s
      timeout: 10s
      retries: 60
    ports:
      - "9044:9042" # Expose native binary CQL port for your apps
    volumes:
      - ../../../DB/cass3:/var/lib/cassandra    # This is the volume that will persist data for cass3 node
      - ../../../DB/cass3-config:/etc/cassandra    # Use your own config files for full control
    environment: *environment
    depends_on:
      cass2:
        # start cass3 only after cass1 is healthy
        condition: service_healthy

Other questions I have

• Is this problem caused by the different ports for the Cassandra node?
• In normal Xandra, there is a setting for default consistency. Is there something similar in cluster mode?

Sorry for the long post. This is my first time opening an issue on GitHub, as I'm not quite sure what information to include.
Please let me know if you want any additional information
Any help or suggestion for anything (not limited to this connection problem) would be greatly appreciated!

[Andrea Leopardi]

Hi @rhit-liuy29, no need to be sorry!

Sorry for the long post. This is my first time opening an issue on GitHub, as I'm not quite sure what information to include.

Welcome to GitHub then

In normal Xandra, there is a setting for default consistency. Is there something similar in cluster mode?

Yes, as documented Xandra.Cluster.start_link/1 also takes all the options from Xandra.start_link/1, including :default_consistency.

Is this problem caused by the different ports for the Cassandra node?

It might be. Your Elixir-side config looks alright at a quick glance. In my experience, however, having a cluster on Docker Compose works within Docker Compose, but you cannot then reliably discover nodes outside of Docker Compose. To try and debug this, try starting a single Cassandra node (through Docker is fine too, just not a cluster) and connect to that node only. Let me know how that goes!

[rhit-liuy29]

Hi @whatyouhide, thank you for the reply! Currently, I'm quite busy with work. I'll get back to you asap.

[Andrea Leopardi]

@rhit-liuy29 ping?

[Andrea Leopardi]

I assume that this has not been an issue anymore, so I’m gonna go ahead and close this. If it's still a problem, feel free to comment here.