Stargate Code Refactoring (v1->v2) #1454
Replies: 1 comment 3 replies
-
|
Very helpful. Please keep us posted on the v2 design decisions and new repo details. thanks |
Beta Was this translation helpful? Give feedback.
-
|
Yes, we will continue to update. We're wrapping up work on our first milestone for v2 - REST API factored out as its own microservice. We hope to cut our first v2 snapshot release soon and publish a blog with what we've learned so far. |
Beta Was this translation helpful? Give feedback.
-
|
May I know the reason why Dropwizard is being used for new REST API microservice instead of modern frameworks like Quarkus or Micronaut ? |
Beta Was this translation helpful? Give feedback.
-
|
We do intend to migrate to a modern framework, but haven't started that work yet. I was chatting with @tatu-at-datastax and @ivansenic yesterday about this very topic, so your timing for asking is very good :). I've started a conversation on the discussions where we'd love to have your input, we're starting with requirements: #1526 |
Beta Was this translation helpful? Give feedback.
-
Stargate Code Refactoring (v1->v2)
This document provides an analysis of the Maven modules in Stargate v1 and proposes some refactorings for Stargate v2.
Background (V1)
The existing Stargate repo consists of about 20 modules, consisting of:
Due to the monolithic nature of the V1 implementation, there are many dependencies across components.
Problems for V2
Stargate V2 effort will decouple the current “Persistence” part (into New Coordinator entity, consisting of Bridge along with Persistence-backed Coordinator) from APIs/Services.
Services will also be decoupled from each other, either living in separate repos, or at least as properly separated share-little sub-modules.
This means that some of the code-level sharing that currently exists needs to be severed: no code shared by being in the same repo (as is the case with REST/Doc API), all sharing via explicit “common” packages.
The current placement of code that is shared by multiple modules is arbitrary, both leading to duplication where functionality is not shared (despite making sense) and sometimes shared in the wrong scope (or sharing things for which sharing adds otherwise unnecessary and awkward dependency -- currently the case for REST/Doc API pair, due to co-location).
Java 9+: No “split” Java packages
Starting with Java 9 (or more accurately, Java module system JPMS), Java packages cannot be split across multiple jars/bundles/modules: something that is allowed up until Java 8.
Currently known problems (Split packages) include:
Dependency to Jersey framework (and maybe DropWizard)
Some of Metrics-registration code as well as wiring relies on Jersey framework. There is obviously a need for wiring for the current implementation. But if we are to consider moving to newer frameworks (especially for APIs/Services), we need to try to minimize them to make moving easier. And at very least we need to be aware what the dependencies are.
Scopes for shared functionality
Considering existing functionality, there seem to be different “scopes” for functionality sharing.
Scopes are defined in the context of V2 view, in which “Persistence” (which refers to the “Coordinator” entity, also including Bridge) is physically separate from Services (APIs). This separation exists mostly as a logical concern in V1.
What follows is a combination of existing state, ideas for grouping shared code along these scopes.
Functionality shared by Persistence implementations
Currently “persistence-common” seems to align with this scope.
The gRPC Bridge component will be packaged in a similar way to the gRPC backend
Functionality shared by HTTP-based Services
Currently not well defined; existing “restapi” contains code that is shared between Doc and REST APIs due to their incidental co-location.
Misc ideas, questions:
restapitypeErrorasApiError(for later extraction to shared package) #1322 )Functionality shared by all Services/APIs (but not Persistence)
Misc ideas:
Functionality shared by Persistence and Services (global)
Currently we have “core” and likely “persistence-api”. Ideally there would be few things we really want to share across all main-level entities but perhaps some cross-cutting concerns should be shared:
Proposed New/Changed/Renamed/Removed Shared Modules
This section provides detailed recommendations by module based on the principles above.
Common modules
Shared by both service-side and persistence-side.
grpc-protomoduleThis module contains the GRPC protos as well as the generated Java client/server stubs. There are repos for other languages:
Current recommendation is to leave these stubs for other languages in their own repos.
coremoduleContains 3 packages
authnzmoduleCore classes for authentication and authorization used in multiple persistence implementations and other locations in v1. Based on design decision for v2 to have authentication performed by the Bridge rather than services, this could potentially be aligned with a persistence-common (see below). No changes required for v2.
health-checkermoduleIncludes health checks for both coordinator and service side modules. Each checker class should be refactored to be as close to where it is used as possible. Each Service will have its own health checker. Checks for OSGi state so in current form are mostly relevant for the coordinator side.
Note also that unit tests are not packaged separately in this module. Unit tests should be moved to an appropriate location in the destination module as part of any refactoring we do.
Coordinator modules (not including Service APIs)
persistence-apimoduleGoal: eliminate the dependence of Service API modules on
persistence-apiRecommend movement of
QueryBuilder(basically, the entireio.stargate.db.querypackage) toservice-common, as this is not used in persistence implementations. (Alternatively,QueryBuildercould be replaced entirely).Io.stargate.dbcontains many classes forming a common language (Table, Column, etc) -> perhaps some of this should move topersistence-common?Io.stargate.db.limiter- only used byrate-limiting-global(which in turn is currently unused) -> consider moving topersistence-common?persistence-commonmoduleThis is indeed only used by other persistence modules. (There may be items we move to here as part of other cleanup?)
This module has a dependency on the cassandra-all module (4.0) which we should try to eliminate, we don’t want a dependency to a specific Cassandra version.
bridge-protomodule (new)Defer creating until we need, probably by copy-paste from
grpc-prototo start.bridgemodule (new)Defer creating until we need, probably by copy paste from
grpc.cqlmoduleNo changes required for v2.
persistence-cassandra-3.11moduleNo changes required for v2.
persistence-cassandra-4.0moduleNo changes required for v2.
persistence-commonmoduleCurrently contains classes that help manage Cassandra-based persistence
TBD - could be a landing place for items we move out of persistence-api?
rate-limiting-globalmoduleContains a single class - the only Implementation of an interface defined in persistence-api. Notes indicate this is not intended for production use. No changes required for v2.
auth-apimoduleThis service provides the authentication endpoint clients use to get credentials. This would continue to be part of the coordinator node because the table-based authentication implementation needs to interact with the selected persistence backend. No changes required for v2.
auth-jwt-servicemoduleProvides implementations of the
AuthenticationServiceandAuthorizationServicedefined inauthnz.This would continue to be part of the coordinator node. No changes required for v2, but I think we should consider refactoring auth handling to support a chain instead of a single implementation. That way you can register your auth provider with the bridge and configure the order instead of picking a particular implementation.
The dependencies on the persistence-api include the
o.a.c.stargate.exceptions.AuthenticationException. We should consider moving this tocore.Also depends on other model classes especially
ResultSet, again candidates for moving to core.auth-table-based-servicemoduleAlternate implementation of the AuthenticationService and AuthorizationService defined in authnz. This would continue to be part of the coordinator node. No changes required for v2. Similar comments as on
auth-jwt-servicemodule apply here.stargate-startermoduleNothing else depends on this, but treating as common because it is currently used to start everything.
Service-side modules
service-commonmodule (new)TBD contents
rest-apimoduleRe-implement
io.stargate.web.restapi.dao.RestDB(previously known asAuthenticatedDb) to call gRPC stub instead ofDataStorefrompersistence-api. Then address any remaining dependencies onpersistence-api.document-apimodule (new)Complete refactoring to separate the document API out of v1.
Re-implement
io.stargate.web.docsapi.DocumentDBto to call gRPC stub instead of DataStore frompersistence-api. Then address any remaining dependencies onpersistence-api.grpcmoduleAt first this can remain in the coordinator. This can be copied as the initial implementation of the bridge. Then re-implement
io.stargate.grpc.service.*to call bridge gRPC stub instead ofPersistencefrompersistence-api. We can then address any remaining dependencies onpersistence-api.Test related modules
persistence-testmoduleThis is used by
persistence-cassandra-3.11,persistence-cassandra-4.0. Common classes for persistence testing. No changes required for v2.testingmoduleIntegration tests. Will require some changes to deployment in order to run Service APIs. Will be a good opportunity to run tests that leverage only the API under test.
testing-servicesmoduleThis provides a self-contained jar that facilitates testing.
Beta Was this translation helpful? Give feedback.
All reactions