Unable to specify Cassandra BroadcastAddress #2825
Comments
|
The setting but the variable is defaulted to the same as the |
|
Thank you for raising the issue. I wanted to check and see if you had looked at the docker compose configurations we provide in the repo. For example this configuration for running Stargate with Cassandra 4: https://github.com/stargate/stargate/blob/main/docker-compose/cassandra-4.0/docker-compose.yml. In this configuration with everything on the same Docker network there is no need to configure listen or broadcast address. If you have a similar deployment perhaps there is no need to configure these properties? In the code that you cite, Stargate does default the broadcast address to the listen address if no broadcast address is provided. This is the expected behavior and emulates what Cassandra itself does. Are you saying that you are specifying the |
|
@jeffreyscarpenter Thank you for the rapid feedback. I looked at the Docker Compose settings on the website, but as you said, they are only for a cluster on the same Docker network. None of the examples (at least as far as I have found) demonstrate the use case across a cluster of multiple physically separated servers, which is the actual use case for us. I agree that the automatic fallback to the listen address is consistent in logic with Cassandra. However, I don't know how to set the broadcast address, I haven't found a corresponding environment variable. Our current setting looks like this: services:
#
# CASSANDRA NODE
#
cassandra:
image: bitnami/cassandra:4.1.3
container_name: cassandra
hostname: cassandra
user: root
# 7000 for inter cluster communication of cassandra
# 9042 for native cql requests. as this node should not get native access (stargate handles this) this should not be opened
ports: [7000:7000,9042:9042]
profiles: [CAS,CAS+API,CAS+KAF,CAS+KAF+API]
volumes:
# data volume
- ./Data/Cassandra:/bitnami/cassandra
environment:
# the name of the cluster
CASSANDRA_CLUSTER_NAME: ${CASSANDRA_CLUSTER_NAME}
# the address other nodes should use to contact this node
CASSANDRA_BROADCAST_ADDRESS: ${CASSANDRA_BROADCAST_ADDRESS}
# the transport post number for inter node communication
CASSANDRA_TRANSPORT_PORT_NUMBER: 7000
# where to fetch the data from. leave empty if this is the first node
CASSANDRA_SEEDS: ${CASSANDRA_SEEDS}
# credentials
CASSANDRA_USER: ${CASSANDRA_USERNAME}
CASSANDRA_PASSWORD: ${CASSANDRA_PASSWORD}
CASSANDRA_PASSWORD_SEEDER: ${CASSANDRA_PASSWORD_SEEDER}
# specify a production ready snitch
CASSANDRA_ENDPOINT_SNITCH: GossipingPropertyFileSnitch
# datacenter and rack specification
CASSANDRA_CFG_RACKDC_DC: ${CASSANDRA_DC}
CASSANDRA_CFG_RACKDC_RACK: ${CASSANDRA_RACK}
# use the local ip if communication inside the sam datacenter
CASSANDRA_CFG_RACKDC_PREFER_LOCAL: true
# performance limitations
MAX_HEAP_SIZE: 2G
HEAP_NEWSIZE: 200M
healthcheck:
# check if the cassandra shell is availabel
test: cqlsh -u cassandra -p cassandra -e 'describe keyspaces'
# after what time should the first healthcheck be run
start_interval: 30s
# which time to tolerate failed responses
start_period: 5m
# check again after
interval: 10s
# timeout after
timeout: 8s
# the number of retries before fail
retries: 3
# restart on failed container
restart: always
#
# CASSANDRA API
#
stargate-coordinator:
image: stargateio/coordinator-4_0:v2.1
container_name: stargate-coordinator
hostname: stargate-coordinator
# 8080 is graphQL interface
# 8081 is auth service
# 8082 is rest interface for crud
# 8084 is health check
# 8180 is document api
# 8090 is gRPC interface
# 7001 is cassandra native transport
# 9043 is cassandra cql service
ports: [8081:8081,8084:8084,8090:8090,7001:7001]
profiles: [CAS+API,CAS+KAF+API]
environment:
# the host (listen_address) will be set to the internal docker ip
# CASSANDRA_HOST:
# the cassandra cluster name
CLUSTER_NAME: ${CASSANDRA_CLUSTER_NAME}
# use local cassandra node
SEED: ${CASSANDRA_BROADCAST_ADDRESS}
# the port for cassandra native transport
SEED_PORT: 7001
# the port for the cql service
CQL_PORT: 9043
# datacenter and rack, as this node
DATACENTER_NAME: ${CASSANDRA_RACK}
RACK_NAME: ${CASSANDRA_DC}
# the major cassandra version
CLUSTER_VERSION: 4.0
ENABLE_AUTH: true
depends_on:
cassandra:
condition: service_healthy
restart: true
healthcheck:
# check if coordinator is available
test: curl -f http://localhost:8084/checker/readiness
# after what time should the first healthcheck be run
start_interval: 30s
# which time to tolerate failed responses
start_period: 5m
# check again after
interval: 10s
# timeout after
timeout: 8s
# the number of retries before fail
retries: 3
# restart on failed container
restart: alwaysIn addition, this is an example .env file. Of course, theese settings change from server to server.. # settings for cassandra
# which ip or address does this server listen to
CASSANDRA_BROADCAST_ADDRESS=192.168.0.18
CASSANDRA_CLUSTER_NAME=test-cassandra-cluster
# datacenter and rack specification
CASSANDRA_RACK=rack02
CASSANDRA_DC=datacenter02
# credentials
CASSANDRA_USERNAME=cassandra
CASSANDRA_PASSWORD=cassandra
# if this is the first cassandra node (ne clsuter) set this to yes, otherwise no
CASSANDRA_PASSWORD_SEEDER=no
# where should the data be fetched from. comma seperated list of other ACTIVE nodes
# if this is only the own ip, this node will automatically start a new cluster
CASSANDRA_SEEDS=192.168.0.238I tried to resolve this Issue in the Stargate Discord beforehand, but sadly there was no response. |
|
@tatu-at-datastax @jeffreyscarpenter Is there any new information here? |
Currently it is not possible to configure the broadcast address for Cassandra. If it is, I couldn't find anything in the documentation.
We are currently using Stargate in a Docker environment. Therefore, there will be servers in the cluster on which both a Cassandra node and a Stargate node will run. Standalone Cassandra also runs in Docker and is configured so that the listen_address gets the Docker internal IP when the Docker container starts. The broadcast address is then set to the public IP of the server. The Cassandra cluster boots up as expected and is able to communicate with each other.
As soon as the Stargate Docker Container is additionally started, the following error occurs:
Nodetool Gossipinfo displays the following information for the cluster:
It can be seen that the standalone Cassandra nodes (192.268.0.18 and 192.168..238) are recognized correctly using the internal IP as the listen address and the public IP as the broadcast address, but the Stargate node is publishing its internal IP. Thus, all other nodes logically get a timeout when they try to call this internal IP.
The text was updated successfully, but these errors were encountered: