FEATURE REQUEST - Support CDC integration

[ErickRamirezDS]

For reference, this was brought up in Stack Overflow question 76284476.

Add the ability to create a containerized Stargate instance running the CDC Java agent for Astra DB.

[aymkhalil]

I'm not familiar with Stargate, but IIUC the user was trying to install cdc agent on coordinator nodes but agent is only required on data storing nodes.

Please note that what's referenced in the question as Datastax CDC and the CDC Agent are two different products - the latter is meant for C*/DSE self managed options.

[SpencerC]

My goal was fully simulate an Astra deployment with CDC enabled by spinning up an ephemeral testcontainers environment each time I run an integration test. I was able to do this by rewriting the "agent" as an OSGi bundle:
ggx-trade/cdc-apache-cassandra@246ffc6

I also had to modify Stargate to export Cassandra and other dependencies from the persistence bundle due to issues with duplicate MBean registrations:
ggx-trade@bfdf0ff

@aymkhalil if there's an easier way LMK, this sideloading approach does feel a bit hacky.

[aymkhalil]

I see. IIUC Stargate sits in front of a "vanilla" C* cluster. The agent is only required on the C* (C3/C4/DSE) cluster side (data storing nodes only) so I'm not sure why it should be bundled any differently when Stargate is used (but I maybe missing some details).

My goal was fully simulate an Astra deployment with CDC enabled by spinning up an ephemeral testcontainers environment

Installation aside, I'd like to clarify that the "agent" based CDC solution does not provide the same promises as the Astra CDC one. Most importantly, it doesn't guarantee that all mutations on the same primary key will show up on the "data topic" - it does guarantee though that the latest state will be available:

E.g. if item with PK X is:

• Created @ T1
• Deleted @ T2
• Created again @ T3

where T1, T2, T3 are in chronological order, there is no guarantee that the DELETE will show up, however the last CREATE will. For more on why this is the case, check "the read before publishing path" on this high level diagram

Having said that, if your case is INSERT only or the mutations are "sparse" in time, you get similar experience to Astra CDC.

Please note that "the read before publishing" operation referenced above is performed by the "connector" that is completely deployed on the Pulsar side of things. The connector also does deduplication, which is required because the agent is deployed on each data node and will be triggered a number of times equal to the replication factor.

Apologies if this has diverged from how to enable CDC agent with Stargate environment, but I wanted make sure you are using the right product for the simulation.

[SpencerC]

Thanks! At the moment, I only care about INSERTs in a local environment, so it seems like this will work.

I'm not sure why it should be bundled any differently when Stargate is used

I tried loading the agent using the -javaagent flag, and spent many hours down a debugging rabbit hole to get to this point. I'm guessing there's a way to build a Stargate-specific version of this that could provide those guarantees. I think it would also be cool if Stargate could provide a streaming gRPC endpoint that could allow users to selectively subscribe to changes without needing to separately manage Pulsar schemas/clients. But that may be another feature request...

[aymkhalil]

@SpencerC you don't need to configure the -javaagent flag on Stargate - only the C* nodes.

Here is an example with testcontainers: https://github.com/datastax/cdc-apache-cassandra/blob/master/testcontainers/src/main/java/com/datastax/testcontainers/cassandra/CassandraContainer.java#L291-L296

To verify, I ran the following experiment locally using the stargate/docker-compose/cassandra-4.0/start_cass_40.sh docker-compose command:

• Built my own C4 image (you could use DSE as well), with cdc agent bundled. I used this Dockerfile:

FROM cassandra:4.0.4
# required to set cdc_enabled=true
COPY cassandra.yaml /etc/cassandra/cassandra.yaml 
# download from https://github.com/datastax/cdc-apache-cassandra/releases/download/v2.2.9/agent-c4-2.2.9-all.jar
COPY agent-c4-2.2.9-all.jar /opt/cdc_agent/cdc-agent.jar 

• Run docker build -t aymkhalil/c40cdc -f c4+cdc.docker . (pushed here if you wanna try it)
• Configure the java agent and use the above image in: start_cass_40.sh :

# do the same for cassandra-2 and cassandra-3
cassandra-1:
    image: aymkhalil:c40cdc
    ...
    environment:
      ...
      - JVM_EXTRA_OPTS=-javaagent:/opt/cdc_agent/cdc-agent.jar

• Run start_cass_40.sh. The log lines to look for in the C* nodes are

2023-05-24 16:42:54 INFO  [main] 2023-05-24 23:42:54,310 DatabaseDescriptor.java:598 - cdc_enabled is true. Starting casssandra node with Change-Data-Capture enabled.
...
2023-05-24 16:54:24 INFO  [main] 2023-05-24 23:54:24,613 Agent.java:60 - Starting CDC agent, cdc_raw_directory=/opt/cassandra/data/cdc_raw

Hope that helps

[SpencerC]

@aymkhalil if I just run the stargate/coordinator-4_0 image it behaves like a working Stargate backend; it responds to CQL queries just like one would expect. Are you saying that in order for CDC to work using the agent approach I need to also spin up a Cassandra cluster for Stargate to actually coordinate? I apologize if this is a basic question, I'm no Cassandra expert!

[aymkhalil]

Ah I get it now - you must be using Stargate with DEVELOPER_MODE=true. I'd go with a separate CDC-enabled Cassandra cluster, for one it works out of the box and for two it is more realistic setup.

Having said that, I understand the convenience of enabling CDC in developer mode but I'd leave it to the Stargate team to chime in. If the decision is to support it, I'm happy to support any efforts required on the CDC agent side (i.e. your OSGi bundle approach or other approaches as recommended by the Stargate team).

[SpencerC]

Thanks! IMHO, the draw of using Astra/Stargate at this point in my company's lifecycle is that we don't need to learn (as an institution) how to support a full Cassandra configuration. Plus this is for an integration test which I'd like my team to be able to run as part of a development workflow, and I feel the Stargate container already takes way too long to startup on its own.