Proposal: Stargate v2.1 including vector search

[jeffreyscarpenter]

As we shared back in March, the Stargate team has been working on a new document-oriented API called the JSON API, along with an extension to the popular Mongoose JS project called stargate-mongoose to make document storage in Cassandra more accessible to JavaScript developers.

In order to implement all of the functionality described in the JSON API Specification, we require additional sorting capabilities in Cassandra. These changes are present in proposed improvements to the Storage Attached Index (SAI) under CEP-7 (which has been merged to the Cassandra trunk and will be part of Cassandra 5.0), and additional improvements targeted for future releases. These improvements are available in the DataStax fork and also in DataStax Astra.

In order to produce the JSON API preview, we needed to make progress on several improvements and deprecations to the main Stargate project which have been discussed previously. We’ve grouped these changes together under a new minor version we’re calling Stargate 2.1, and have updated the JSON API and stargate-mongoose repositories to begin using the v2.1 release series. Stargate 2.1 consists of the following:

• New vector-enabled persistence module (DSE-next persistence backend #2669) - if you’ve been tracking the progress of Cassandra 5.0, you’ll be aware of some exciting changes to Cassandra, including support for vector search, ACID transactions, and the SAI improvements referenced above. For the 2.1 release we are adding a “DSE-next” persistence module that incorporates these features. (We are also exploring the possibility of adding a Cassandra 4.1 persistence module (Support Cassandra 4.1 as a persistence backend #2311), but this will not be vector-enabled.)

• API Support for Vector Search - we're adding vector search capabilities on the JSON, REST, and gRPC APIs that will be available when vector-enabled persistence is used.

• Removal of support for Cassandra 3.11 persistence module (removing Cassandra 3.11 support #2614) - the Apache project has announced plans for support for Cassandra 3.11 to continue until the release of Cassandra 5.0, originally thought to be around May 2023. While the release of 5.0 has now been delayed, we’ve decided to continue with the plan we shared previously to deprecate support for Cassandra 3.11 in Stargate.

• Update of coordinator to Java 11 (Update Coordinator to JDK 11 #2624) - The removal of the Cassandra 3.11 persistence module from the Stargate codebase enables us to upgrade the base Java version used for the coordinator node from Java 8 to Java 11, which is required by Cassandra 4.0. We’re also investigating the possibility of removing usage of the OSGi framework and packaging the coordinator as a Quarkus application, as we did with the Stargate v2 API implementations.

• Removal of legacy HTTP API implementations (Remove REST and GraphQL APIs from coordinator #2640) - although Stargate v2 contained fully backward compatible implementations of the REST, Docs, and GraphQL APIs as independent microservices, we’ve preserved the option to re-enable the old implementations within the coordinator to facilitate migration from Stargate v1 to v2. With the 2.1 release, we’re removing these implementations from the coordinator codebase.

• Support for CQL protocol v5 (Support protocol v5  #770) - the addition of support for this new protocol version added in Cassandra 4.0 will allow Stargate CQL users to use the latest protocol features and improvements. Applications that are using the latest drivers will be able to take advantage of this seamlessly, as the drivers negotiate the protocol version used automatically.

We have started a v2.1 feature branch containing these changes, and are publishing v2.1 alpha/beta releases as the work matures.