Skip to content
This repository has been archived by the owner on Mar 16, 2022. It is now read-only.

Native-image Cassandra smoke test is flaky #527

Open
pvlugter opened this issue Feb 2, 2021 · 2 comments
Open

Native-image Cassandra smoke test is flaky #527

pvlugter opened this issue Feb 2, 2021 · 2 comments

Comments

@pvlugter
Copy link
Member

pvlugter commented Feb 2, 2021

The native-image smoke test for Cassandra is often failing and too unreliable. This test will be disabled until we can address this.

Example logs for failed test 
#!/bin/bash -eo pipefail
bin/run-java-eventsourced-shopping-cart-test.sh Cassandra --no-build

=== Running java-eventsourced-shopping-cart test ===

statefulstore.cloudstate.io/cassandra created
statefulservice.cloudstate.io/shopping-cart-cassandra created

Waiting for deployment to be created...
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
shopping-cart-cassandra   0/1     1            0           1s

Waiting for deployment to be ready...
Waiting for deployment "shopping-cart-cassandra" rollout to finish: 0 of 1 updated replicas are available...
deployment "shopping-cart-cassandra" successfully rolled out
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
shopping-cart-cassandra   1/1     1            1           8s

Scaling deployment...
statefulservice.cloudstate.io/shopping-cart-cassandra scaled
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
shopping-cart-cassandra   1/3     1            1           8s

Waiting for deployment to be ready...
Waiting for deployment "shopping-cart-cassandra" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "shopping-cart-cassandra" rollout to finish: 2 of 3 updated replicas are available...
deployment "shopping-cart-cassandra" successfully rolled out
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
shopping-cart-cassandra   3/3     3            3           23s
service/shopping-cart-cassandra-node-port exposed

Testing eventsourced shopping cart test1 ...
Expected '{"items":[]}'
But got 'The server was not able to produce a timely response to your request.
Please try again in a short while!'
Failed:

=== Operator logs ===

2021-02-02T00:03:21.517Z	INFO	setup	using config	{"file": "/etc/config/config.yaml"}
2021-02-02T00:03:22.172Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": "127.0.0.1:8080"}
2021-02-02T00:03:22.172Z	INFO	controller-runtime.webhook	registering webhook	{"path": "/inject-v1-pod"}
2021-02-02T00:03:22.172Z	INFO	setup	starting manager
I0202 00:03:22.174217       1 leaderelection.go:242] attempting to acquire leader lease  cloudstate-system/cloudstate-leader-election-helper...
2021-02-02T00:03:22.174Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
2021-02-02T00:03:22.175Z	INFO	controller-runtime.webhook.webhooks	starting webhook server
2021-02-02T00:03:22.175Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
2021-02-02T00:03:22.175Z	INFO	controller-runtime.webhook	serving webhook server	{"host": "", "port": 9443}
2021-02-02T00:03:22.217Z	INFO	controller-runtime.certwatcher	Starting certificate watcher
I0202 00:03:22.222430       1 leaderelection.go:252] successfully acquired lease cloudstate-system/cloudstate-leader-election-helper
2021-02-02T00:03:22.222Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"ConfigMap","namespace":"cloudstate-system","name":"cloudstate-leader-election-helper","uid":"2f295404-3bbd-4c05-bea8-39e7d27c1e81","apiVersion":"v1","resourceVersion":"715"}, "reason": "LeaderElection", "message": "cloudstate-controller-manager-648c764f7b-84nsh_0288eb0a-1691-41b7-8a73-8a9ca7b6da8b became leader"}
2021-02-02T00:03:22.222Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "source": "kind source: /, Kind="}
2021-02-02T00:03:22.222Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulStore", "controller": "statefulstore", "source": "kind source: /, Kind="}
2021-02-02T00:03:22.323Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "source": "kind source: /, Kind="}
2021-02-02T00:03:22.323Z	INFO	controller	Starting Controller	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulStore", "controller": "statefulstore"}
2021-02-02T00:03:22.517Z	INFO	controller	Starting workers	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulStore", "controller": "statefulstore", "worker count": 1}
2021-02-02T00:03:22.517Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "source": "kind source: /, Kind="}
2021-02-02T00:03:22.618Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "source": "kind source: /, Kind="}
2021-02-02T00:03:22.718Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "source": "kind source: /, Kind="}
2021-02-02T00:03:22.819Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "source": "kind source: /, Kind="}
2021-02-02T00:03:22.919Z	INFO	controller	Starting Controller	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice"}
2021-02-02T00:03:22.919Z	INFO	controller	Starting workers	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "worker count": 1}
2021-02-02T00:03:27.020Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/inject-v1-pod", "UID": "e5137699-440f-41eb-a033-6c4a242cab6c", "kind": "/v1, Kind=Pod", "resource": {"group":"","version":"v1","resource":"pods"}}
2021-02-02T00:03:27.026Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/inject-v1-pod", "code": 200, "reason": "not a stateful service, not injecting", "UID": "e5137699-440f-41eb-a033-6c4a242cab6c", "allowed": true}
2021-02-02T00:04:45.218Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulStore", "controller": "statefulstore", "name": "cassandra", "namespace": "default"}
2021-02-02T00:04:45.218Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulStore", "controller": "statefulstore", "name": "cassandra", "namespace": "default"}
2021-02-02T00:04:45.543Z	INFO	controllers.StatefulService	Updated status	{"statefulservice": "default/shopping-cart-cassandra", "replicas": 0, "summary": "Unavailable"}
2021-02-02T00:04:45.548Z	INFO	controllers.StatefulService	Created deployment	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:45.548Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:45.549Z	DEBUG	controllers.StatefulService	Inspecting status	{"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 1, "status.updatedReplicas": 0, "status.availableReplicas": 0}
2021-02-02T00:04:45.558Z	INFO	controllers.StatefulService	Updated status	{"statefulservice": "default/shopping-cart-cassandra", "replicas": 0, "summary": "UpdateInProgress"}
2021-02-02T00:04:45.560Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/inject-v1-pod", "UID": "574b962d-2e4d-4872-8cb0-16d964f7166c", "kind": "/v1, Kind=Pod", "resource": {"group":"","version":"v1","resource":"pods"}}
2021-02-02T00:04:45.617Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/inject-v1-pod", "code": 200, "reason": "", "UID": "574b962d-2e4d-4872-8cb0-16d964f7166c", "allowed": true}
2021-02-02T00:04:45.618Z	DEBUG	controllers.StatefulService	No change in deployment	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:45.618Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:45.619Z	DEBUG	controllers.StatefulService	Inspecting status	{"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 1, "status.updatedReplicas": 0, "status.availableReplicas": 0}
2021-02-02T00:04:45.619Z	DEBUG	controllers.StatefulService	No status update	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:45.621Z	DEBUG	controllers.StatefulService	No change in deployment	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:45.621Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:45.646Z	DEBUG	controllers.StatefulService	Inspecting status	{"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 1, "status.updatedReplicas": 1, "status.availableReplicas": 0}
2021-02-02T00:04:45.652Z	INFO	controllers.StatefulService	Updated status	{"statefulservice": "default/shopping-cart-cassandra", "replicas": 0, "summary": "Unavailable"}
2021-02-02T00:04:45.717Z	DEBUG	controllers.StatefulService	No change in deployment	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:45.717Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:45.718Z	DEBUG	controllers.StatefulService	Inspecting status	{"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 1, "status.updatedReplicas": 1, "status.availableReplicas": 0}
2021-02-02T00:04:45.718Z	DEBUG	controllers.StatefulService	No status update	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:45.719Z	DEBUG	controllers.StatefulService	No change in deployment	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:45.719Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:53.465Z	DEBUG	controllers.StatefulService	Inspecting status	{"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 1, "status.updatedReplicas": 1, "status.availableReplicas": 1}
2021-02-02T00:04:53.517Z	INFO	controllers.StatefulService	Updated status	{"statefulservice": "default/shopping-cart-cassandra", "replicas": 1, "summary": "Ready"}
2021-02-02T00:04:53.519Z	DEBUG	controllers.StatefulService	No change in deployment	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.519Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:53.520Z	DEBUG	controllers.StatefulService	Inspecting status	{"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 1, "status.updatedReplicas": 1, "status.availableReplicas": 1}
2021-02-02T00:04:53.520Z	DEBUG	controllers.StatefulService	No status update	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.522Z	DEBUG	controllers.StatefulService	No change in deployment	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.522Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:53.676Z	DEBUG	controllers.StatefulService	Inspecting status	{"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 1, "status.updatedReplicas": 1, "status.availableReplicas": 1}
2021-02-02T00:04:53.676Z	DEBUG	controllers.StatefulService	No status update	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.679Z	DEBUG	controllers.StatefulService	changed	{"statefulservice": "default/shopping-cart-cassandra", "type": "deployment", "patchResult": "\nPatch: {\"metadata\":{\"annotations\":{\"banzaicloud.com/last-applied\":\"{\\\"metadata\\\":{\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"user-function\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"cloudstate-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"shopping-cart-cassandra\\\"},\\\"name\\\":\\\"shopping-cart-cassandra\\\",\\\"namespace\\\":\\\"default\\\",\\\"ownerReferences\\\":[{\\\"apiVersion\\\":\\\"cloudstate.io/v1alpha1\\\",\\\"blockOwnerDeletion\\\":true,\\\"controller\\\":true,\\\"kind\\\":\\\"StatefulService\\\",\\\"name\\\":\\\"shopping-cart-cassandra\\\",\\\"uid\\\":\\\"28abd19d-86cc-4692-bf2a-015a54be1cd1\\\"}]},\\\"spec\\\":{\\\"replicas\\\":3,\\\"selector\\\":{\\\"matchLabels\\\":{\\\"cloudstate.io/stateful-service\\\":\\\"shopping-cart-cassandra\\\"}},\\\"strategy\\\":{},\\\"template\\\":{\\\"metadata\\\":{\\\"annotations\\\":{\\\"cassandra.cloudstate.io/keyspace\\\":\\\"shoppingcart\\\",\\\"cloudstate.io/enabled\\\":\\\"true\\\",\\\"cloudstate.io/stateful-service-config\\\":\\\"ss-cfg-shopping-cart-cassandra\\\",\\\"cloudstate.io/stateful-store\\\":\\\"cassandra\\\"},\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"user-function\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"cloudstate-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"shopping-cart-cassandra\\\",\\\"cloudstate.io/stateful-service\\\":\\\"shopping-cart-cassandra\\\"}},\\\"spec\\\":{\\\"containers\\\":[{\\\"image\\\":\\\"cloudstateio/java-eventsourced-shopping-cart:latest\\\",\\\"imagePullPolicy\\\":\\\"Never\\\",\\\"name\\\":\\\"user-function\\\",\\\"resources\\\":{}}]}}},\\\"status\\\":{}}\"}},\"spec\":{\"replicas\":3}} \nCurrent: {\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"banzaicloud.com/last-applied\":\"{\\\"metadata\\\":{\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"user-function\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"cloudstate-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"shopping-cart-cassandra\\\"},\\\"name\\\":\\\"shopping-cart-cassandra\\\",\\\"namespace\\\":\\\"default\\\",\\\"ownerReferences\\\":[{\\\"apiVersion\\\":\\\"cloudstate.io/v1alpha1\\\",\\\"blockOwnerDeletion\\\":true,\\\"controller\\\":true,\\\"kind\\\":\\\"StatefulService\\\",\\\"name\\\":\\\"shopping-cart-cassandra\\\",\\\"uid\\\":\\\"28abd19d-86cc-4692-bf2a-015a54be1cd1\\\"}]},\\\"spec\\\":{\\\"replicas\\\":1,\\\"selector\\\":{\\\"matchLabels\\\":{\\\"cloudstate.io/stateful-service\\\":\\\"shopping-cart-cassandra\\\"}},\\\"strategy\\\":{},\\\"template\\\":{\\\"metadata\\\":{\\\"annotations\\\":{\\\"cassandra.cloudstate.io/keyspace\\\":\\\"shoppingcart\\\",\\\"cloudstate.io/enabled\\\":\\\"true\\\",\\\"cloudstate.io/stateful-service-config\\\":\\\"ss-cfg-shopping-cart-cassandra\\\",\\\"cloudstate.io/stateful-store\\\":\\\"cassandra\\\"},\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"user-function\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"cloudstate-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"shopping-cart-cassandra\\\",\\\"cloudstate.io/stateful-service\\\":\\\"shopping-cart-cassandra\\\"}},\\\"spec\\\":{\\\"containers\\\":[{\\\"image\\\":\\\"cloudstateio/java-eventsourced-shopping-cart:latest\\\",\\\"imagePullPolicy\\\":\\\"Never\\\",\\\"name\\\":\\\"user-function\\\",\\\"resources\\\":{}}]}}},\\\"status\\\":{}}\",\"deployment.kubernetes.io/revision\":\"1\"},\"creationTimestamp\":\"2021-02-02T00:04:45Z\",\"generation\":1,\"labels\":{\"app.kubernetes.io/component\":\"user-function\",\"app.kubernetes.io/managed-by\":\"cloudstate-operator\",\"app.kubernetes.io/name\":\"shopping-cart-cassandra\"},\"managedFields\":[{\"apiVersion\":\"apps/v1\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:banzaicloud.com/last-applied\":{}},\"f:labels\":{\".\":{},\"f:app.kubernetes.io/component\":{},\"f:app.kubernetes.io/managed-by\":{},\"f:app.kubernetes.io/name\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"28abd19d-86cc-4692-bf2a-015a54be1cd1\\\"}\":{\".\":{},\"f:apiVersion\":{},\"f:blockOwnerDeletion\":{},\"f:controller\":{},\"f:kind\":{},\"f:name\":{},\"f:uid\":{}}}},\"f:spec\":{\"f:progressDeadlineSeconds\":{},\"f:replicas\":{},\"f:revisionHistoryLimit\":{},\"f:selector\":{\"f:matchLabels\":{\".\":{},\"f:cloudstate.io/stateful-service\":{}}},\"f:strategy\":{\"f:rollingUpdate\":{\".\":{},\"f:maxSurge\":{},\"f:maxUnavailable\":{}},\"f:type\":{}},\"f:template\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:cassandra.cloudstate.io/keyspace\":{},\"f:cloudstate.io/enabled\":{},\"f:cloudstate.io/stateful-service-config\":{},\"f:cloudstate.io/stateful-store\":{}},\"f:labels\":{\".\":{},\"f:app.kubernetes.io/component\":{},\"f:app.kubernetes.io/managed-by\":{},\"f:app.kubernetes.io/name\":{},\"f:cloudstate.io/stateful-service\":{}}},\"f:spec\":{\"f:containers\":{\"k:{\\\"name\\\":\\\"user-function\\\"}\":{\".\":{},\"f:image\":{},\"f:imagePullPolicy\":{},\"f:name\":{},\"f:resources\":{},\"f:terminationMessagePath\":{},\"f:terminationMessagePolicy\":{}}},\"f:dnsPolicy\":{},\"f:restartPolicy\":{},\"f:schedulerName\":{},\"f:securityContext\":{},\"f:terminationGracePeriodSeconds\":{}}}}},\"manager\":\"manager\",\"operation\":\"Update\",\"time\":\"2021-02-02T00:04:45Z\"},{\"apiVersion\":\"apps/v1\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:deployment.kubernetes.io/revision\":{}}},\"f:status\":{\"f:availableReplicas\":{},\"f:conditions\":{\".\":{},\"k:{\\\"type\\\":\\\"Available\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:lastUpdateTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"Progressing\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:lastUpdateTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}}},\"f:observedGeneration\":{},\"f:readyReplicas\":{},\"f:replicas\":{},\"f:updatedReplicas\":{}}},\"manager\":\"kube-controller-manager\",\"operation\":\"Update\",\"time\":\"2021-02-02T00:04:53Z\"}],\"name\":\"shopping-cart-cassandra\",\"namespace\":\"default\",\"ownerReferences\":[{\"apiVersion\":\"cloudstate.io/v1alpha1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"StatefulService\",\"name\":\"shopping-cart-cassandra\",\"uid\":\"28abd19d-86cc-4692-bf2a-015a54be1cd1\"}],\"resourceVersion\":\"1014\",\"selfLink\":\"/apis/apps/v1/namespaces/default/deployments/shopping-cart-cassandra\",\"uid\":\"459e45de-a4a5-485a-ab9b-f296869812f3\"},\"spec\":{\"progressDeadlineSeconds\":600,\"replicas\":1,\"revisionHistoryLimit\":10,\"selector\":{\"matchLabels\":{\"cloudstate.io/stateful-service\":\"shopping-cart-cassandra\"}},\"strategy\":{\"rollingUpdate\":{\"maxSurge\":\"25%\",\"maxUnavailable\":\"25%\"},\"type\":\"RollingUpdate\"},\"template\":{\"metadata\":{\"annotations\":{\"cassandra.cloudstate.io/keyspace\":\"shoppingcart\",\"cloudstate.io/enabled\":\"true\",\"cloudstate.io/stateful-service-config\":\"ss-cfg-shopping-cart-cassandra\",\"cloudstate.io/stateful-store\":\"cassandra\"},\"labels\":{\"app.kubernetes.io/component\":\"user-function\",\"app.kubernetes.io/managed-by\":\"cloudstate-operator\",\"app.kubernetes.io/name\":\"shopping-cart-cassandra\",\"cloudstate.io/stateful-service\":\"shopping-cart-cassandra\"}},\"spec\":{\"containers\":[{\"image\":\"cloudstateio/java-eventsourced-shopping-cart:latest\",\"imagePullPolicy\":\"Never\",\"name\":\"user-function\",\"resources\":{},\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\"}],\"dnsPolicy\":\"ClusterFirst\",\"restartPolicy\":\"Always\",\"schedulerName\":\"default-scheduler\",\"securityContext\":{},\"terminationGracePeriodSeconds\":30}}},\"status\":{\"availableReplicas\":1,\"conditions\":[{\"lastTransitionTime\":\"2021-02-02T00:04:53Z\",\"lastUpdateTime\":\"2021-02-02T00:04:53Z\",\"message\":\"Deployment has minimum availability.\",\"reason\":\"MinimumReplicasAvailable\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2021-02-02T00:04:45Z\",\"lastUpdateTime\":\"2021-02-02T00:04:53Z\",\"message\":\"ReplicaSet \\\"shopping-cart-cassandra-65569f85cd\\\" has successfully progressed.\",\"reason\":\"NewReplicaSetAvailable\",\"status\":\"True\",\"type\":\"Progressing\"}],\"observedGeneration\":1,\"readyReplicas\":1,\"replicas\":1,\"updatedReplicas\":1}}\nModified: {\"metadata\":{\"annotations\":{\"banzaicloud.com/last-applied\":\"{\\\"metadata\\\":{\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"user-function\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"cloudstate-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"shopping-cart-cassandra\\\"},\\\"name\\\":\\\"shopping-cart-cassandra\\\",\\\"namespace\\\":\\\"default\\\",\\\"ownerReferences\\\":[{\\\"apiVersion\\\":\\\"cloudstate.io/v1alpha1\\\",\\\"blockOwnerDeletion\\\":true,\\\"controller\\\":true,\\\"kind\\\":\\\"StatefulService\\\",\\\"name\\\":\\\"shopping-cart-cassandra\\\",\\\"uid\\\":\\\"28abd19d-86cc-4692-bf2a-015a54be1cd1\\\"}]},\\\"spec\\\":{\\\"replicas\\\":3,\\\"selector\\\":{\\\"matchLabels\\\":{\\\"cloudstate.io/stateful-service\\\":\\\"shopping-cart-cassandra\\\"}},\\\"strategy\\\":{},\\\"template\\\":{\\\"metadata\\\":{\\\"annotations\\\":{\\\"cassandra.cloudstate.io/keyspace\\\":\\\"shoppingcart\\\",\\\"cloudstate.io/enabled\\\":\\\"true\\\",\\\"cloudstate.io/stateful-service-config\\\":\\\"ss-cfg-shopping-cart-cassandra\\\",\\\"cloudstate.io/stateful-store\\\":\\\"cassandra\\\"},\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"user-function\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"cloudstate-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"shopping-cart-cassandra\\\",\\\"cloudstate.io/stateful-service\\\":\\\"shopping-cart-cassandra\\\"}},\\\"spec\\\":{\\\"containers\\\":[{\\\"image\\\":\\\"cloudstateio/java-eventsourced-shopping-cart:latest\\\",\\\"imagePullPolicy\\\":\\\"Never\\\",\\\"name\\\":\\\"user-function\\\",\\\"resources\\\":{}}]}}},\\\"status\\\":{}}\"},\"labels\":{\"app.kubernetes.io/component\":\"user-function\",\"app.kubernetes.io/managed-by\":\"cloudstate-operator\",\"app.kubernetes.io/name\":\"shopping-cart-cassandra\"},\"name\":\"shopping-cart-cassandra\",\"namespace\":\"default\",\"ownerReferences\":[{\"apiVersion\":\"cloudstate.io/v1alpha1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"StatefulService\",\"name\":\"shopping-cart-cassandra\",\"uid\":\"28abd19d-86cc-4692-bf2a-015a54be1cd1\"}]},\"spec\":{\"replicas\":3,\"selector\":{\"matchLabels\":{\"cloudstate.io/stateful-service\":\"shopping-cart-cassandra\"}},\"strategy\":{},\"template\":{\"metadata\":{\"annotations\":{\"cassandra.cloudstate.io/keyspace\":\"shoppingcart\",\"cloudstate.io/enabled\":\"true\",\"cloudstate.io/stateful-service-config\":\"ss-cfg-shopping-cart-cassandra\",\"cloudstate.io/stateful-store\":\"cassandra\"},\"labels\":{\"app.kubernetes.io/component\":\"user-function\",\"app.kubernetes.io/managed-by\":\"cloudstate-operator\",\"app.kubernetes.io/name\":\"shopping-cart-cassandra\",\"cloudstate.io/stateful-service\":\"shopping-cart-cassandra\"}},\"spec\":{\"containers\":[{\"image\":\"cloudstateio/java-eventsourced-shopping-cart:latest\",\"imagePullPolicy\":\"Never\",\"name\":\"user-function\",\"resources\":{}}]}}},\"status\":{}}\nOriginal: {\"metadata\":{\"labels\":{\"app.kubernetes.io/component\":\"user-function\",\"app.kubernetes.io/managed-by\":\"cloudstate-operator\",\"app.kubernetes.io/name\":\"shopping-cart-cassandra\"},\"name\":\"shopping-cart-cassandra\",\"namespace\":\"default\",\"ownerReferences\":[{\"apiVersion\":\"cloudstate.io/v1alpha1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"StatefulService\",\"name\":\"shopping-cart-cassandra\",\"uid\":\"28abd19d-86cc-4692-bf2a-015a54be1cd1\"}]},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"cloudstate.io/stateful-service\":\"shopping-cart-cassandra\"}},\"strategy\":{},\"template\":{\"metadata\":{\"annotations\":{\"cassandra.cloudstate.io/keyspace\":\"shoppingcart\",\"cloudstate.io/enabled\":\"true\",\"cloudstate.io/stateful-service-config\":\"ss-cfg-shopping-cart-cassandra\",\"cloudstate.io/stateful-store\":\"cassandra\"},\"labels\":{\"app.kubernetes.io/component\":\"user-function\",\"app.kubernetes.io/managed-by\":\"cloudstate-operator\",\"app.kubernetes.io/name\":\"shopping-cart-cassandra\",\"cloudstate.io/stateful-service\":\"shopping-cart-cassandra\"}},\"spec\":{\"containers\":[{\"image\":\"cloudstateio/java-eventsourced-shopping-cart:latest\",\"imagePullPolicy\":\"Never\",\"name\":\"user-function\",\"resources\":{}}]}}},\"status\":{}}\n"}
2021-02-02T00:04:53.684Z	INFO	controllers.StatefulService	Updated deployment	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.684Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:53.685Z	DEBUG	controllers.StatefulService	Inspecting status	{"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 1, "status.availableReplicas": 1}
2021-02-02T00:04:53.702Z	INFO	controllers.StatefulService	Updated status	{"statefulservice": "default/shopping-cart-cassandra", "replicas": 1, "summary": "UpdateInProgress"}
2021-02-02T00:04:53.704Z	DEBUG	controllers.StatefulService	No change in deployment	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.704Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:53.720Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/inject-v1-pod", "UID": "3b3c3740-345a-4a90-84fc-2e890ad2cda8", "kind": "/v1, Kind=Pod", "resource": {"group":"","version":"v1","resource":"pods"}}
2021-02-02T00:04:53.721Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/inject-v1-pod", "code": 200, "reason": "", "UID": "3b3c3740-345a-4a90-84fc-2e890ad2cda8", "allowed": true}
2021-02-02T00:04:53.723Z	DEBUG	controllers.StatefulService	Inspecting status	{"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 1, "status.availableReplicas": 1}
2021-02-02T00:04:53.723Z	DEBUG	controllers.StatefulService	No status update	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.726Z	DEBUG	controllers.StatefulService	No change in deployment	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.726Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:53.727Z	DEBUG	controllers.StatefulService	Inspecting status	{"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 1, "status.availableReplicas": 1}
2021-02-02T00:04:53.729Z	DEBUG	controllers.StatefulService	No status update	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.732Z	DEBUG	controller-runtime.webhook.webhooks	received request	{"webhook": "/inject-v1-pod", "UID": "54b8cc51-9755-4154-ab50-18ca941a8a7e", "kind": "/v1, Kind=Pod", "resource": {"group":"","version":"v1","resource":"pods"}}
2021-02-02T00:04:53.817Z	DEBUG	controller-runtime.webhook.webhooks	wrote response	{"webhook": "/inject-v1-pod", "code": 200, "reason": "", "UID": "54b8cc51-9755-4154-ab50-18ca941a8a7e", "allowed": true}
2021-02-02T00:04:53.819Z	DEBUG	controllers.StatefulService	No change in deployment	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.819Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:53.854Z	DEBUG	controllers.StatefulService	Inspecting status	{"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 3, "status.availableReplicas": 1}
2021-02-02T00:04:53.919Z	INFO	controllers.StatefulService	Updated status	{"statefulservice": "default/shopping-cart-cassandra", "replicas": 1, "summary": "PartiallyReady"}
2021-02-02T00:04:53.920Z	DEBUG	controllers.StatefulService	No change in deployment	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.920Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:53.921Z	DEBUG	controllers.StatefulService	Inspecting status	{"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 3, "status.availableReplicas": 1}
2021-02-02T00:04:53.921Z	DEBUG	controllers.StatefulService	No status update	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.923Z	DEBUG	controllers.StatefulService	No change in deployment	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.923Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:05:00.926Z	DEBUG	controllers.StatefulService	Inspecting status	{"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 3, "status.availableReplicas": 1}
2021-02-02T00:05:00.926Z	DEBUG	controllers.StatefulService	No status update	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:05:00.928Z	DEBUG	controllers.StatefulService	No change in deployment	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:05:00.928Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:05:08.418Z	DEBUG	controllers.StatefulService	Inspecting status	{"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 3, "status.availableReplicas": 2}
2021-02-02T00:05:08.424Z	INFO	controllers.StatefulService	Updated status	{"statefulservice": "default/shopping-cart-cassandra", "replicas": 2, "summary": "PartiallyReady"}
2021-02-02T00:05:08.425Z	DEBUG	controllers.StatefulService	No change in deployment	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:05:08.425Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:05:08.428Z	DEBUG	controllers.StatefulService	Inspecting status	{"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 3, "status.availableReplicas": 2}
2021-02-02T00:05:08.428Z	DEBUG	controllers.StatefulService	No status update	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:05:08.429Z	DEBUG	controllers.StatefulService	No change in deployment	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:05:08.429Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:05:08.607Z	DEBUG	controllers.StatefulService	Inspecting status	{"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 3, "status.availableReplicas": 3}
2021-02-02T00:05:08.614Z	INFO	controllers.StatefulService	Updated status	{"statefulservice": "default/shopping-cart-cassandra", "replicas": 3, "summary": "Ready"}
2021-02-02T00:05:08.615Z	DEBUG	controllers.StatefulService	No change in deployment	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:05:08.615Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:05:08.616Z	DEBUG	controllers.StatefulService	Inspecting status	{"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 3, "status.availableReplicas": 3}
2021-02-02T00:05:08.616Z	DEBUG	controllers.StatefulService	No status update	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:05:08.617Z	DEBUG	controllers.StatefulService	No change in deployment	{"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:05:08.617Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}

=== Deployment description ===

Name:                   shopping-cart-cassandra
Namespace:              default
CreationTimestamp:      Tue, 02 Feb 2021 00:04:45 +0000
Labels:                 app.kubernetes.io/component=user-function
                        app.kubernetes.io/managed-by=cloudstate-operator
                        app.kubernetes.io/name=shopping-cart-cassandra
Annotations:            banzaicloud.com/last-applied:
                          {"metadata":{"labels":{"app.kubernetes.io/component":"user-function","app.kubernetes.io/managed-by":"cloudstate-operator","app.kubernetes....
                        deployment.kubernetes.io/revision: 1
Selector:               cloudstate.io/stateful-service=shopping-cart-cassandra
Replicas:               3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:       app.kubernetes.io/component=user-function
                app.kubernetes.io/managed-by=cloudstate-operator
                app.kubernetes.io/name=shopping-cart-cassandra
                cloudstate.io/stateful-service=shopping-cart-cassandra
  Annotations:  cassandra.cloudstate.io/keyspace: shoppingcart
                cloudstate.io/enabled: true
                cloudstate.io/stateful-service-config: ss-cfg-shopping-cart-cassandra
                cloudstate.io/stateful-store: cassandra
  Containers:
   user-function:
    Image:        cloudstateio/java-eventsourced-shopping-cart:latest
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   shopping-cart-cassandra-65569f85cd (3/3 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  44s   deployment-controller  Scaled up replica set shopping-cart-cassandra-65569f85cd to 1
  Normal  ScalingReplicaSet  36s   deployment-controller  Scaled up replica set shopping-cart-cassandra-65569f85cd to 3

=== Pods description ===

Name:         cassandra-0
Namespace:    default
Priority:     0
Node:         default-235cd9d9-df1d-4cc5-b459-28892cef4871/10.128.1.60
Start Time:   Tue, 02 Feb 2021 00:03:27 +0000
Labels:       app=cassandra
              chart=cassandra-5.6.1
              controller-revision-hash=cassandra-67cf96f4b8
              heritage=Helm
              release=cassandra
              statefulset.kubernetes.io/pod-name=cassandra-0
Annotations:  <none>
Status:       Running
IP:           172.17.0.7
IPs:
  IP:           172.17.0.7
Controlled By:  StatefulSet/cassandra
Containers:
  cassandra:
    Container ID:  docker://ef7252fa690d037a6a83ee7c8334e33256c2ad57a742fb1c4eda6d9e5d39b58f
    Image:         docker.io/bitnami/cassandra:3.11.7-debian-10-r4
    Image ID:      docker-pullable://bitnami/cassandra@sha256:c864afbe56596553c72a3babf9c72158e89241a7e786eddbc2fa2551b56403ef
    Ports:         7000/TCP, 7001/TCP, 7199/TCP, 9042/TCP, 9160/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    Command:
      bash
      -ec
      if [[ $HOSTNAME =~ (.*)-0$ ]]; then
        echo "Setting node as password seeder"
        export CASSANDRA_PASSWORD_SEEDER=yes
      else
        # Only node 0 will execute the startup initdb scripts
        export CASSANDRA_IGNORE_INITDB_SCRIPTS=1
      fi
      /entrypoint.sh /run.sh
      
    State:          Running
      Started:      Tue, 02 Feb 2021 00:03:40 +0000
    Ready:          True
    Restart Count:  0
    Liveness:       exec [/bin/sh -c nodetool status] delay=60s timeout=5s period=30s #success=1 #failure=5
    Readiness:      exec [/bin/sh -c nodetool status | grep -E "^UN\s+${POD_IP}"] delay=60s timeout=5s period=10s #success=1 #failure=5
    Environment:
      BITNAMI_DEBUG:              false
      CASSANDRA_CLUSTER_NAME:     cassandra
      CASSANDRA_SEEDS:            cassandra-0.cassandra-headless.default.svc.cluster.local
      CASSANDRA_PASSWORD:         <set to the key 'cassandra-password' in secret 'cassandra'>  Optional: false
      POD_IP:                      (v1:status.podIP)
      CASSANDRA_USER:             cassandra
      CASSANDRA_NUM_TOKENS:       256
      CASSANDRA_DATACENTER:       dc1
      CASSANDRA_ENDPOINT_SNITCH:  SimpleSnitch
      CASSANDRA_ENDPOINT_SNITCH:  SimpleSnitch
      CASSANDRA_RACK:             rack1
      CASSANDRA_ENABLE_RPC:       true
    Mounts:
      /bitnami/cassandra from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wd7lz (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  default-token-wd7lz:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-wd7lz
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From                                                   Message
  ----    ------     ----  ----                                                   -------
  Normal  Scheduled  2m2s  default-scheduler                                      Successfully assigned default/cassandra-0 to default-235cd9d9-df1d-4cc5-b459-28892cef4871
  Normal  Pulling    2m2s  kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Pulling image "docker.io/bitnami/cassandra:3.11.7-debian-10-r4"
  Normal  Pulled     109s  kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Successfully pulled image "docker.io/bitnami/cassandra:3.11.7-debian-10-r4"
  Normal  Created    109s  kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Created container cassandra
  Normal  Started    109s  kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Started container cassandra


Name:         shopping-cart-cassandra-65569f85cd-dxpbr
Namespace:    default
Priority:     0
Node:         default-235cd9d9-df1d-4cc5-b459-28892cef4871/10.128.1.60
Start Time:   Tue, 02 Feb 2021 00:04:53 +0000
Labels:       app.kubernetes.io/component=user-function
              app.kubernetes.io/managed-by=cloudstate-operator
              app.kubernetes.io/name=shopping-cart-cassandra
              cloudstate.io/stateful-service=shopping-cart-cassandra
              pod-template-hash=65569f85cd
Annotations:  cassandra.cloudstate.io/keyspace: shoppingcart
              cloudstate.io/enabled: true
              cloudstate.io/stateful-service-config: ss-cfg-shopping-cart-cassandra
              cloudstate.io/stateful-store: cassandra
              traffic.sidecar.istio.io/excludeInboundPorts: 2552,8558
              traffic.sidecar.istio.io/excludeOutboundPorts: 2552,8558
Status:       Running
IP:           172.17.0.10
IPs:
  IP:           172.17.0.10
Controlled By:  ReplicaSet/shopping-cart-cassandra-65569f85cd
Containers:
  user-function:
    Container ID:   docker://062c42b378755454c4ca7be16a1f818bf62cfca5203bce8330133b9ac75dc40a
    Image:          cloudstateio/java-eventsourced-shopping-cart:latest
    Image ID:       docker://sha256:3df7437cc0f3bce3c4a5c9ff7293a884ce2e61a29cadbee171a849f204c5c35f
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 02 Feb 2021 00:04:54 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      PORT:  8080
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wd7lz (ro)
  cloudstate-sidecar:
    Container ID:   docker://4b0f0cc559a8517acd98ab2309470d97987af5a41d68a942e99fa910ddc1b213
    Image:          cloudstateio/cloudstate-proxy-native-cassandra:latest
    Image ID:       docker://sha256:0ba59b5fc83c69f703d7703cb92630c031944712d61d91efd14ec6056f249559
    Ports:          8013/TCP, 9099/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Tue, 02 Feb 2021 00:04:54 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  512Mi
    Requests:
      cpu:      400m
      memory:   512Mi
    Liveness:   http-get http://:8558/alive delay=0s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:8558/ready delay=0s timeout=1s period=10s #success=1 #failure=5
    Environment:
      USER_FUNCTION_PORT:         8080
      METRICS_PORT:               9099
      REMOTING_PORT:              2552
      MANAGEMENT_PORT:            8558
      HTTP_PORT:                  8013
      SELECTOR_LABEL:             cloudstate.io/stateful-service
      SELECTOR_LABEL_VALUE:       shopping-cart-cassandra
      REQUIRED_CONTACT_POINT_NR:  1
      CASSANDRA_CONTACT_POINTS:   cassandra.default.svc.cluster.local
      CASSANDRA_KEYSPACE:         shoppingcart
      CASSANDRA_USERNAME:         <set to the key 'username' in secret 'cassandra-credentials'>  Optional: false
      CASSANDRA_PASSWORD:         <set to the key 'password' in secret 'cassandra-credentials'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wd7lz (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-wd7lz:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-wd7lz
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age   From                                                   Message
  ----     ------     ----  ----                                                   -------
  Normal   Scheduled  36s   default-scheduler                                      Successfully assigned default/shopping-cart-cassandra-65569f85cd-dxpbr to default-235cd9d9-df1d-4cc5-b459-28892cef4871
  Normal   Pulled     35s   kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Container image "cloudstateio/java-eventsourced-shopping-cart:latest" already present on machine
  Normal   Created    35s   kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Created container user-function
  Normal   Started    35s   kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Started container user-function
  Normal   Pulled     35s   kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Container image "cloudstateio/cloudstate-proxy-native-cassandra:latest" already present on machine
  Normal   Created    35s   kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Created container cloudstate-sidecar
  Normal   Started    35s   kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Started container cloudstate-sidecar
  Warning  Unhealthy  31s   kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Readiness probe failed: HTTP probe failed with statuscode: 500


Name:         shopping-cart-cassandra-65569f85cd-g4vq5
Namespace:    default
Priority:     0
Node:         default-235cd9d9-df1d-4cc5-b459-28892cef4871/10.128.1.60
Start Time:   Tue, 02 Feb 2021 00:04:45 +0000
Labels:       app.kubernetes.io/component=user-function
              app.kubernetes.io/managed-by=cloudstate-operator
              app.kubernetes.io/name=shopping-cart-cassandra
              cloudstate.io/stateful-service=shopping-cart-cassandra
              pod-template-hash=65569f85cd
Annotations:  cassandra.cloudstate.io/keyspace: shoppingcart
              cloudstate.io/enabled: true
              cloudstate.io/stateful-service-config: ss-cfg-shopping-cart-cassandra
              cloudstate.io/stateful-store: cassandra
              traffic.sidecar.istio.io/excludeInboundPorts: 2552,8558
              traffic.sidecar.istio.io/excludeOutboundPorts: 2552,8558
Status:       Running
IP:           172.17.0.8
IPs:
  IP:           172.17.0.8
Controlled By:  ReplicaSet/shopping-cart-cassandra-65569f85cd
Containers:
  user-function:
    Container ID:   docker://b238c07e01234ef592fd85ca9f08d3902bc447c222911429e09a8044991461cc
    Image:          cloudstateio/java-eventsourced-shopping-cart:latest
    Image ID:       docker://sha256:3df7437cc0f3bce3c4a5c9ff7293a884ce2e61a29cadbee171a849f204c5c35f
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 02 Feb 2021 00:04:46 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      PORT:  8080
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wd7lz (ro)
  cloudstate-sidecar:
    Container ID:   docker://a97aca57bb76b2fe1e9913d5879dddab0f108e8fd0cee85933780d108bcffd7b
    Image:          cloudstateio/cloudstate-proxy-native-cassandra:latest
    Image ID:       docker://sha256:0ba59b5fc83c69f703d7703cb92630c031944712d61d91efd14ec6056f249559
    Ports:          8013/TCP, 9099/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Tue, 02 Feb 2021 00:04:46 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  512Mi
    Requests:
      cpu:      400m
      memory:   512Mi
    Liveness:   http-get http://:8558/alive delay=0s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:8558/ready delay=0s timeout=1s period=10s #success=1 #failure=5
    Environment:
      USER_FUNCTION_PORT:         8080
      METRICS_PORT:               9099
      REMOTING_PORT:              2552
      MANAGEMENT_PORT:            8558
      HTTP_PORT:                  8013
      SELECTOR_LABEL:             cloudstate.io/stateful-service
      SELECTOR_LABEL_VALUE:       shopping-cart-cassandra
      REQUIRED_CONTACT_POINT_NR:  1
      CASSANDRA_CONTACT_POINTS:   cassandra.default.svc.cluster.local
      CASSANDRA_KEYSPACE:         shoppingcart
      CASSANDRA_USERNAME:         <set to the key 'username' in secret 'cassandra-credentials'>  Optional: false
      CASSANDRA_PASSWORD:         <set to the key 'password' in secret 'cassandra-credentials'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wd7lz (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-wd7lz:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-wd7lz
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From                                                   Message
  ----    ------     ----  ----                                                   -------
  Normal  Scheduled  44s   default-scheduler                                      Successfully assigned default/shopping-cart-cassandra-65569f85cd-g4vq5 to default-235cd9d9-df1d-4cc5-b459-28892cef4871
  Normal  Pulled     43s   kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Container image "cloudstateio/java-eventsourced-shopping-cart:latest" already present on machine
  Normal  Created    43s   kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Created container user-function
  Normal  Started    43s   kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Started container user-function
  Normal  Pulled     43s   kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Container image "cloudstateio/cloudstate-proxy-native-cassandra:latest" already present on machine
  Normal  Created    43s   kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Created container cloudstate-sidecar
  Normal  Started    43s   kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Started container cloudstate-sidecar


Name:         shopping-cart-cassandra-65569f85cd-q6jd5
Namespace:    default
Priority:     0
Node:         default-235cd9d9-df1d-4cc5-b459-28892cef4871/10.128.1.60
Start Time:   Tue, 02 Feb 2021 00:04:53 +0000
Labels:       app.kubernetes.io/component=user-function
              app.kubernetes.io/managed-by=cloudstate-operator
              app.kubernetes.io/name=shopping-cart-cassandra
              cloudstate.io/stateful-service=shopping-cart-cassandra
              pod-template-hash=65569f85cd
Annotations:  cassandra.cloudstate.io/keyspace: shoppingcart
              cloudstate.io/enabled: true
              cloudstate.io/stateful-service-config: ss-cfg-shopping-cart-cassandra
              cloudstate.io/stateful-store: cassandra
              traffic.sidecar.istio.io/excludeInboundPorts: 2552,8558
              traffic.sidecar.istio.io/excludeOutboundPorts: 2552,8558
Status:       Running
IP:           172.17.0.9
IPs:
  IP:           172.17.0.9
Controlled By:  ReplicaSet/shopping-cart-cassandra-65569f85cd
Containers:
  user-function:
    Container ID:   docker://2f1dc02b9a6254404e8bcce1489d2c4c7b02ea01f3fb0c7729922cdfbf68ce7e
    Image:          cloudstateio/java-eventsourced-shopping-cart:latest
    Image ID:       docker://sha256:3df7437cc0f3bce3c4a5c9ff7293a884ce2e61a29cadbee171a849f204c5c35f
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 02 Feb 2021 00:04:54 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      PORT:  8080
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wd7lz (ro)
  cloudstate-sidecar:
    Container ID:   docker://b30cac0bd4cd0027d7773b3c401900438232779e633ae2f3cac54d0f3cd040c5
    Image:          cloudstateio/cloudstate-proxy-native-cassandra:latest
    Image ID:       docker://sha256:0ba59b5fc83c69f703d7703cb92630c031944712d61d91efd14ec6056f249559
    Ports:          8013/TCP, 9099/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Tue, 02 Feb 2021 00:04:54 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  512Mi
    Requests:
      cpu:      400m
      memory:   512Mi
    Liveness:   http-get http://:8558/alive delay=0s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:8558/ready delay=0s timeout=1s period=10s #success=1 #failure=5
    Environment:
      USER_FUNCTION_PORT:         8080
      METRICS_PORT:               9099
      REMOTING_PORT:              2552
      MANAGEMENT_PORT:            8558
      HTTP_PORT:                  8013
      SELECTOR_LABEL:             cloudstate.io/stateful-service
      SELECTOR_LABEL_VALUE:       shopping-cart-cassandra
      REQUIRED_CONTACT_POINT_NR:  1
      CASSANDRA_CONTACT_POINTS:   cassandra.default.svc.cluster.local
      CASSANDRA_KEYSPACE:         shoppingcart
      CASSANDRA_USERNAME:         <set to the key 'username' in secret 'cassandra-credentials'>  Optional: false
      CASSANDRA_PASSWORD:         <set to the key 'password' in secret 'cassandra-credentials'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wd7lz (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-wd7lz:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-wd7lz
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age   From                                                   Message
  ----     ------     ----  ----                                                   -------
  Normal   Scheduled  36s   default-scheduler                                      Successfully assigned default/shopping-cart-cassandra-65569f85cd-q6jd5 to default-235cd9d9-df1d-4cc5-b459-28892cef4871
  Normal   Pulled     35s   kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Container image "cloudstateio/java-eventsourced-shopping-cart:latest" already present on machine
  Normal   Created    35s   kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Created container user-function
  Normal   Started    35s   kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Started container user-function
  Normal   Pulled     35s   kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Container image "cloudstateio/cloudstate-proxy-native-cassandra:latest" already present on machine
  Normal   Created    35s   kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Created container cloudstate-sidecar
  Normal   Started    35s   kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Started container cloudstate-sidecar
  Warning  Unhealthy  31s   kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871  Readiness probe failed: HTTP probe failed with statuscode: 500

=== Proxy logs ===

2021-02-02 00:04:54.921 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2021-02-02 00:04:54.944 WARN akka.actor.ActorSystemImpl - Failed to load JFR flight recorder, falling back to noop. Exception: java.lang.ClassNotFoundException: akka.remote.artery.jfr.JFRRemotingFlightRecorder
2021-02-02 00:04:54.949 INFO akka.remote.artery.tcp.ArteryTcpTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cloudstate-proxy@172.17.0.10:2552] with UID [5053097932859970139]
2021-02-02 00:04:54.950 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.10:2552] - Starting up, Akka version [2.6.9] ...
2021-02-02 00:04:54.951 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.10:2552] - Started up successfully
2021-02-02 00:04:54.957 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.10:2552] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining
2021-02-02 00:04:54.958 INFO akka.cluster.sbr.SplitBrainResolver - SBR started. Config: stableAfter: 20000 ms, strategy: KeepMajority, selfUniqueAddress: UniqueAddress(akka://cloudstate-proxy@172.17.0.10:2552,5053097932859970139), selfDc: default
2021-02-02 00:04:54.968 INFO akka.actor.ActorSystemImpl - Starting Cloudstate Proxy version [0.6.0-5-f0446998] running on [Substrate VM GraalVM 20.3.1 Java 11]
2021-02-02 00:04:54.968 INFO akka.actor.ActorSystemImpl - Registering SIGTERM handler...
2021-02-02 00:04:54.968 INFO akka.management.internal.HealthChecksImpl - Loading readiness checks [(cloudstate,io.cloudstate.proxy.HealthCheckReady)]
2021-02-02 00:04:54.968 INFO akka.management.internal.HealthChecksImpl - Loading liveness checks [(cloudstate,io.cloudstate.proxy.HealthCheckLive)]
2021-02-02 00:04:54.969 INFO akka.management.scaladsl.AkkaManagement - Binding Akka Management (HTTP) endpoint to: 172.17.0.10:8558
2021-02-02 00:04:54.969 INFO akka.management.scaladsl.AkkaManagement - Including HTTP management routes for ClusterBootstrap
2021-02-02 00:04:54.969 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Using self contact point address: http://172.17.0.10:8558
2021-02-02 00:04:54.969 INFO akka.management.scaladsl.AkkaManagement - Including HTTP management routes for HealthCheckRoutes
2021-02-02 00:04:54.975 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Initiating bootstrap procedure using kubernetes-api method...
2021-02-02 00:04:54.975 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Bootstrap using `akka.discovery` method: kubernetes-api
2021-02-02 00:04:54.975 INFO akka.management.scaladsl.AkkaManagement - Bound Akka Management (HTTP) endpoint to: 172.17.0.10:8558
2021-02-02 00:04:54.978 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Locating service members. Using discovery [akka.discovery.kubernetes.KubernetesApiServiceDiscovery], join decider [akka.management.cluster.bootstrap.LowestAddressJoinDecider], scheme [http]
2021-02-02 00:04:54.978 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Looking up [Lookup(shopping-cart-cassandra,None,Some(tcp))]
2021-02-02 00:04:54.978 INFO akka.discovery.kubernetes.KubernetesApiServiceDiscovery - Querying for pods with label selector: [cloudstate.io/stateful-service=shopping-cart-cassandra]. Namespace: [default]. Port: [None]
2021-02-02 00:04:55.025 INFO akka.actor.ActorSystemImpl - Prometheus exporter started on /0.0.0.0:9099
2021-02-02 00:04:55.052 ERROR akka.actor.OneForOneStrategy - UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at io.grpc.Status.asRuntimeException(Status.java:533)
	at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
	at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(Thread.java:834)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:8080
Caused by: java.net.ConnectException: Connection refused
	at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_net_ConnectException_2_0002e_0003cinit_0003e_00028Ljava_lang_String_2_00029V(JNIJavaCallWrappers.java:0)
	at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
	at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
	at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:834)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
2021-02-02 00:04:55.073 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Located service members based on: [Lookup(shopping-cart-cassandra,None,Some(tcp))]: [ResolvedTarget(172-17-0-8.default.pod.cluster.local,None,Some(/172.17.0.8))], filtered to [172-17-0-8.default.pod.cluster.local:0]
2021-02-02 00:04:55.089 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Contact point [akka://cloudstate-proxy@172.17.0.8:2552] returned [1] seed-nodes [akka://cloudstate-proxy@172.17.0.8:2552]
2021-02-02 00:04:55.089 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Joining [akka://cloudstate-proxy@172.17.0.10:2552] to existing cluster [akka://cloudstate-proxy@172.17.0.8:2552]
2021-02-02 00:04:55.089 INFO akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message [akka.management.cluster.bootstrap.internal.BootstrapCoordinator$Protocol$BootstrappingCompleted$] from Actor[akka://cloudstate-proxy/system/bootstrapCoordinator#1156786877] to Actor[akka://cloudstate-proxy/deadLetters] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cloudstate-proxy/deadLetters] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2021-02-02 00:04:55.249 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.10:2552] - Received InitJoinAck message from [Actor[akka://cloudstate-proxy@172.17.0.8:2552/system/cluster/core/daemon#-754755589]] to [akka://cloudstate-proxy@172.17.0.10:2552]
2021-02-02 00:04:55.375 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.10:2552] - Welcome from [akka://cloudstate-proxy@172.17.0.8:2552]
2021-02-02 00:04:55.978 INFO akka.cluster.sbr.SplitBrainResolver - This node is now the leader responsible for taking SBR decisions among the reachable nodes (more leaders may exist).
2021-02-02 00:04:56.257 ERROR akka.actor.OneForOneStrategy - UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at io.grpc.Status.asRuntimeException(Status.java:533)
	at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
	at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(Thread.java:834)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:8080
Caused by: java.net.ConnectException: Connection refused
	at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_net_ConnectException_2_0002e_0003cinit_0003e_00028Ljava_lang_String_2_00029V(JNIJavaCallWrappers.java:0)
	at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
	at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
	at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:834)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
2021-02-02 00:04:56.281 INFO akka.actor.EmptyLocalActorRef - Message [akka.cluster.ddata.Replicator$Internal$Status] from Actor[akka://cloudstate-proxy@172.17.0.8:2552/system/sharding/replicator#-1039544820] to Actor[akka://cloudstate-proxy/system/sharding/replicator] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://cloudstate-proxy/system/sharding/replicator] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2021-02-02 00:04:56.995 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.10:2552] - is the new leader among reachable nodes (more leaders may exist)
2021-02-02 00:04:58.300 INFO akka.actor.EmptyLocalActorRef - Message [akka.cluster.ddata.Replicator$Internal$Status] from Actor[akka://cloudstate-proxy@172.17.0.8:2552/system/sharding/replicator#-1039544820] to Actor[akka://cloudstate-proxy/system/sharding/replicator] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://cloudstate-proxy/system/sharding/replicator] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2021-02-02 00:04:58.391 DEBUG io.cloudstate.proxy.HealthCheckReady - Error performing server manager readiness check
akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka://cloudstate-proxy/), Path(/user/server-manager-supervisor/server-manager)]
	at akka.actor.ActorSelection.$anonfun$resolveOne$1(ActorSelection.scala:74)
	at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:447)
	at scala.concurrent.ExecutionContext$parasitic$.execute(ExecutionContext.scala:164)
	at scala.concurrent.impl.Promise$Transformation.submitWithValue(Promise.scala:392)
	at scala.concurrent.impl.Promise$DefaultPromise.submitWithValue(Promise.scala:302)
	at scala.concurrent.impl.Promise$DefaultPromise.dispatchOrAddCallbacks(Promise.scala:276)
	at scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:180)
	at akka.actor.ActorSelection.resolveOne(ActorSelection.scala:72)
	at io.cloudstate.proxy.HealthCheckReady.check(CloudStateProxyMain.scala:48)
	at io.cloudstate.proxy.HealthCheckReady.apply(CloudStateProxyMain.scala:61)
	at io.cloudstate.proxy.HealthCheckReady.apply(CloudStateProxyMain.scala:38)
	at scala.util.Try$.apply(Try.scala:210)
	at akka.management.internal.HealthChecksImpl.runCheck(HealthChecksImpl.scala:177)
	at akka.management.internal.HealthChecksImpl.$anonfun$check$2(HealthChecksImpl.scala:195)
	at scala.collection.immutable.List.map(List.scala:246)
	at scala.collection.immutable.List.map(List.scala:79)
	at akka.management.internal.HealthChecksImpl.check(HealthChecksImpl.scala:185)
	at akka.management.internal.HealthChecksImpl.readyResult(HealthChecksImpl.scala:149)
	at akka.management.HealthCheckRoutes.$anonfun$routes$3(HealthCheckRoutes.scala:46)
	at akka.http.scaladsl.server.directives.FutureDirectives.$anonfun$onComplete$2(FutureDirectives.scala:37)
	at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResult$2(BasicDirectives.scala:68)
	at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:161)
	at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRequestContext$2(BasicDirectives.scala:45)
	at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:161)
	at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47)
	at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
	at akka.http.scaladsl.util.FastFuture$.transformWith$extension(FastFuture.scala:45)
	at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
	at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
	at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
	at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47)
	at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
	at akka.http.scaladsl.util.FastFuture$.transformWith$extension(FastFuture.scala:45)
	at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
	at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
	at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRequestContext$2(BasicDirectives.scala:45)
	at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:161)
	at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResultWith$2(BasicDirectives.scala:74)
	at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:161)
	at akka.http.scaladsl.server.directives.ExecutionDirectives.$anonfun$handleExceptions$2(ExecutionDirectives.scala:32)
	at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:161)
	at akka.http.scaladsl.server.Route$.$anonfun$asyncHandler$1(Route.scala:86)
	at akka.stream.impl.fusing.MapAsyncUnordered$$anon$31.onPush(Ops.scala:1400)
	at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:541)
	at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:423)
	at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:625)
	at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:502)
	at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:600)
	at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:769)
	at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:784)
	at akka.actor.Actor.aroundReceive(Actor.scala:537)
	at akka.actor.Actor.aroundReceive$(Actor.scala:535)
	at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:691)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:577)
	at akka.actor.ActorCell.invoke(ActorCell.scala:547)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
	at akka.dispatch.Mailbox.run(Mailbox.scala:231)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
2021-02-02 00:04:58.392 INFO akka.management.internal.HealthChecksImpl - Check [io.cloudstate.proxy.HealthCheckReady] not ok
2021-02-02 00:04:58.676 INFO akka.actor.EmptyLocalActorRef - Message [akka.cluster.ddata.Replicator$Internal$Write] from Actor[akka://cloudstate-proxy@172.17.0.8:2552/system/sharding/replicator/$c#-1166386490] to Actor[akka://cloudstate-proxy/system/sharding/replicator] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://cloudstate-proxy/system/sharding/replicator] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2021-02-02 00:04:58.981 INFO io.cloudstate.proxy.EntityDiscoveryManager - Received EntitySpec from user function with info: ServiceInfo(,,OpenJDK Runtime Environment 11.0.10+9,cloudstate-java-support,0.6.0-4-f0446998,0,2,UnknownFieldSet(Map()))
2021-02-02 00:04:58.981 DEBUG io.cloudstate.proxy.eventsourced.EventSourcedSupportFactory - Starting EventSourcedEntity for eventsourced-shopping-cart
2021-02-02 00:04:58.982 INFO EventingManager - Eventing support turned off in configuration
2021-02-02 00:04:58.982 INFO akka.cluster.singleton.ClusterSingletonManager - ClusterSingletonManager state change [Start -> Younger]
2021-02-02 00:04:58.982 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Post(/cart/{user_id}/items/add),*,,Vector(),UnknownFieldSet(Map()))]
2021-02-02 00:04:58.983 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Post(/cart/{user_id}/items/{product_id}/remove),,,Vector(),UnknownFieldSet(Map()))]
2021-02-02 00:04:58.983 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Get(/carts/{user_id}),,,Vector(HttpRule(,Get(/carts/{user_id}/items),,items,Vector(),UnknownFieldSet(Map()))),UnknownFieldSet(Map()))]
2021-02-02 00:04:58.984 DEBUG io.cloudstate.proxy.EntityDiscoveryManager - Starting gRPC proxy
2021-02-02 00:04:58.987 INFO io.cloudstate.proxy.EntityDiscoveryManager - CloudState proxy online at /0.0.0.0:8013
2021-02-02 00:04:46.680 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2021-02-02 00:04:46.691 WARN akka.actor.ActorSystemImpl - Failed to load JFR flight recorder, falling back to noop. Exception: java.lang.ClassNotFoundException: akka.remote.artery.jfr.JFRRemotingFlightRecorder
2021-02-02 00:04:46.697 INFO akka.remote.artery.tcp.ArteryTcpTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cloudstate-proxy@172.17.0.8:2552] with UID [1154167314093528637]
2021-02-02 00:04:46.697 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Starting up, Akka version [2.6.9] ...
2021-02-02 00:04:46.698 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Started up successfully
2021-02-02 00:04:46.699 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining
2021-02-02 00:04:46.699 INFO akka.cluster.sbr.SplitBrainResolver - SBR started. Config: stableAfter: 20000 ms, strategy: KeepMajority, selfUniqueAddress: UniqueAddress(akka://cloudstate-proxy@172.17.0.8:2552,1154167314093528637), selfDc: default
2021-02-02 00:04:46.716 INFO akka.actor.ActorSystemImpl - Starting Cloudstate Proxy version [0.6.0-5-f0446998] running on [Substrate VM GraalVM 20.3.1 Java 11]
2021-02-02 00:04:46.716 INFO akka.actor.ActorSystemImpl - Registering SIGTERM handler...
2021-02-02 00:04:46.716 INFO akka.management.internal.HealthChecksImpl - Loading readiness checks [(cloudstate,io.cloudstate.proxy.HealthCheckReady)]
2021-02-02 00:04:46.716 INFO akka.management.internal.HealthChecksImpl - Loading liveness checks [(cloudstate,io.cloudstate.proxy.HealthCheckLive)]
2021-02-02 00:04:46.719 INFO akka.management.scaladsl.AkkaManagement - Binding Akka Management (HTTP) endpoint to: 172.17.0.8:8558
2021-02-02 00:04:46.719 INFO akka.management.scaladsl.AkkaManagement - Including HTTP management routes for ClusterBootstrap
2021-02-02 00:04:46.719 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Using self contact point address: http://172.17.0.8:8558
2021-02-02 00:04:46.719 INFO akka.management.scaladsl.AkkaManagement - Including HTTP management routes for HealthCheckRoutes
2021-02-02 00:04:46.724 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Initiating bootstrap procedure using kubernetes-api method...
2021-02-02 00:04:46.724 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Bootstrap using `akka.discovery` method: kubernetes-api
2021-02-02 00:04:46.724 INFO akka.management.scaladsl.AkkaManagement - Bound Akka Management (HTTP) endpoint to: 172.17.0.8:8558
2021-02-02 00:04:46.729 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Locating service members. Using discovery [akka.discovery.kubernetes.KubernetesApiServiceDiscovery], join decider [akka.management.cluster.bootstrap.LowestAddressJoinDecider], scheme [http]
2021-02-02 00:04:46.729 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Looking up [Lookup(shopping-cart-cassandra,None,Some(tcp))]
2021-02-02 00:04:46.729 INFO akka.discovery.kubernetes.KubernetesApiServiceDiscovery - Querying for pods with label selector: [cloudstate.io/stateful-service=shopping-cart-cassandra]. Namespace: [default]. Port: [None]
2021-02-02 00:04:46.731 INFO akka.actor.ActorSystemImpl - Prometheus exporter started on /0.0.0.0:9099
2021-02-02 00:04:46.765 ERROR akka.actor.OneForOneStrategy - UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at io.grpc.Status.asRuntimeException(Status.java:533)
	at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
	at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(Thread.java:834)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:8080
Caused by: java.net.ConnectException: Connection refused
	at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_net_ConnectException_2_0002e_0003cinit_0003e_00028Ljava_lang_String_2_00029V(JNIJavaCallWrappers.java:0)
	at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
	at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
	at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:834)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
2021-02-02 00:04:46.786 INFO akka.discovery.kubernetes.KubernetesApiServiceDiscovery - No targets found from pod list. Is the correct port name configured? Current configuration: [None]. Ports on pods: [Set(ContainerPort(Some(user-port),8080), ContainerPort(Some(grpc-http-proxy),8013), ContainerPort(Some(cs-metrics),9099))]
2021-02-02 00:04:46.786 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Located service members based on: [Lookup(shopping-cart-cassandra,None,Some(tcp))]: [], filtered to []
2021-02-02 00:04:47.740 INFO akka.management.cluster.bootstrap.LowestAddressJoinDecider - Discovered [0] contact points, confirmed [0], which is less than the required [1], retrying
2021-02-02 00:04:47.811 ERROR akka.actor.OneForOneStrategy - UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at io.grpc.Status.asRuntimeException(Status.java:533)
	at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
	at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(Thread.java:834)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:8080
Caused by: java.net.ConnectException: Connection refused
	at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_net_ConnectException_2_0002e_0003cinit_0003e_00028Ljava_lang_String_2_00029V(JNIJavaCallWrappers.java:0)
	at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
	at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
	at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:834)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
2021-02-02 00:04:47.930 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Looking up [Lookup(shopping-cart-cassandra,None,Some(tcp))]
2021-02-02 00:04:47.930 INFO akka.discovery.kubernetes.KubernetesApiServiceDiscovery - Querying for pods with label selector: [cloudstate.io/stateful-service=shopping-cart-cassandra]. Namespace: [default]. Port: [None]
2021-02-02 00:04:47.934 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Located service members based on: [Lookup(shopping-cart-cassandra,None,Some(tcp))]: [ResolvedTarget(172-17-0-8.default.pod.cluster.local,None,Some(/172.17.0.8))], filtered to [172-17-0-8.default.pod.cluster.local:0]
2021-02-02 00:04:47.955 INFO akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes - Bootstrap request from 172.17.0.8:48894: Contact Point returning 0 seed-nodes []
2021-02-02 00:04:47.956 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Contact point [akka://cloudstate-proxy@172.17.0.8:2552] returned [0] seed-nodes []
2021-02-02 00:04:48.980 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Looking up [Lookup(shopping-cart-cassandra,None,Some(tcp))]
2021-02-02 00:04:48.981 INFO akka.discovery.kubernetes.KubernetesApiServiceDiscovery - Querying for pods with label selector: [cloudstate.io/stateful-service=shopping-cart-cassandra]. Namespace: [default]. Port: [None]
2021-02-02 00:04:48.992 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Located service members based on: [Lookup(shopping-cart-cassandra,None,Some(tcp))]: [ResolvedTarget(172-17-0-8.default.pod.cluster.local,None,Some(/172.17.0.8))], filtered to [172-17-0-8.default.pod.cluster.local:0]
2021-02-02 00:04:49.080 INFO akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes - Bootstrap request from 172.17.0.8:48894: Contact Point returning 0 seed-nodes []
2021-02-02 00:04:49.080 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Contact point [akka://cloudstate-proxy@172.17.0.8:2552] returned [0] seed-nodes []
2021-02-02 00:04:49.739 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Initiating new cluster, self-joining [akka://cloudstate-proxy@172.17.0.8:2552]. Other nodes are expected to locate this cluster via continued contact-point probing.
2021-02-02 00:04:49.740 INFO akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message [akka.management.cluster.bootstrap.internal.BootstrapCoordinator$Protocol$BootstrappingCompleted$] from Actor[akka://cloudstate-proxy/system/bootstrapCoordinator#1480424403] to Actor[akka://cloudstate-proxy/deadLetters] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cloudstate-proxy/deadLetters] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2021-02-02 00:04:49.740 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Node [akka://cloudstate-proxy@172.17.0.8:2552] is JOINING itself (with roles [dc-default]) and forming new cluster
2021-02-02 00:04:49.740 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - is the new leader among reachable nodes (more leaders may exist)
2021-02-02 00:04:49.740 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Leader is moving node [akka://cloudstate-proxy@172.17.0.8:2552] to [Up]
2021-02-02 00:04:49.740 INFO akka.cluster.sbr.SplitBrainResolver - This node is now the leader responsible for taking SBR decisions among the reachable nodes (more leaders may exist).
2021-02-02 00:04:50.235 INFO io.cloudstate.proxy.EntityDiscoveryManager - Received EntitySpec from user function with info: ServiceInfo(,,OpenJDK Runtime Environment 11.0.10+9,cloudstate-java-support,0.6.0-4-f0446998,0,2,UnknownFieldSet(Map()))
2021-02-02 00:04:50.236 DEBUG io.cloudstate.proxy.eventsourced.EventSourcedSupportFactory - Starting EventSourcedEntity for eventsourced-shopping-cart
2021-02-02 00:04:50.237 INFO EventingManager - Eventing support turned off in configuration
2021-02-02 00:04:50.238 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Post(/cart/{user_id}/items/add),*,,Vector(),UnknownFieldSet(Map()))]
2021-02-02 00:04:50.238 INFO akka.cluster.singleton.ClusterSingletonManager - Singleton manager starting singleton actor [akka://cloudstate-proxy/system/sharding/eventsourced-shopping-cartCoordinator/singleton]
2021-02-02 00:04:50.238 INFO akka.cluster.singleton.ClusterSingletonManager - ClusterSingletonManager state change [Start -> Oldest]
2021-02-02 00:04:50.238 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Post(/cart/{user_id}/items/{product_id}/remove),,,Vector(),UnknownFieldSet(Map()))]
2021-02-02 00:04:50.238 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Get(/carts/{user_id}),,,Vector(HttpRule(,Get(/carts/{user_id}/items),,items,Vector(),UnknownFieldSet(Map()))),UnknownFieldSet(Map()))]
2021-02-02 00:04:50.238 INFO akka.cluster.sharding.DDataShardCoordinator - ShardCoordinator was moved to the active state State(Map())
2021-02-02 00:04:50.239 DEBUG io.cloudstate.proxy.EntityDiscoveryManager - Starting gRPC proxy
2021-02-02 00:04:50.241 INFO io.cloudstate.proxy.EntityDiscoveryManager - CloudState proxy online at /0.0.0.0:8013
2021-02-02 00:04:55.086 INFO akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes - Bootstrap request from 172.17.0.9:50312: Contact Point returning 1 seed-nodes [akka://cloudstate-proxy@172.17.0.8:2552]
2021-02-02 00:04:55.088 INFO akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes - Bootstrap request from 172.17.0.10:58948: Contact Point returning 1 seed-nodes [akka://cloudstate-proxy@172.17.0.8:2552]
2021-02-02 00:04:55.125 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Received InitJoin message from [Actor[akka://cloudstate-proxy@172.17.0.10:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#188025117]] to [akka://cloudstate-proxy@172.17.0.8:2552]
2021-02-02 00:04:55.125 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Sending InitJoinAck message from node [akka://cloudstate-proxy@172.17.0.8:2552] to [Actor[akka://cloudstate-proxy@172.17.0.10:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#188025117]] (version [2.6.9])
2021-02-02 00:04:55.249 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Received InitJoin message from [Actor[akka://cloudstate-proxy@172.17.0.9:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#933739104]] to [akka://cloudstate-proxy@172.17.0.8:2552]
2021-02-02 00:04:55.249 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Sending InitJoinAck message from node [akka://cloudstate-proxy@172.17.0.8:2552] to [Actor[akka://cloudstate-proxy@172.17.0.9:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#933739104]] (version [2.6.9])
2021-02-02 00:04:55.374 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Node [akka://cloudstate-proxy@172.17.0.10:2552] is JOINING, roles [dc-default]
2021-02-02 00:04:55.379 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Node [akka://cloudstate-proxy@172.17.0.9:2552] is JOINING, roles [dc-default]
2021-02-02 00:04:55.859 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Leader is moving node [akka://cloudstate-proxy@172.17.0.10:2552] to [Up]
2021-02-02 00:04:55.860 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - Leader is moving node [akka://cloudstate-proxy@172.17.0.9:2552] to [Up]
2021-02-02 00:04:55.860 INFO akka.cluster.sbr.SplitBrainResolver - This node is not the leader any more and not responsible for taking SBR decisions.
2021-02-02 00:04:56.880 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.8:2552] - is no longer leader
2021-02-02 00:05:03.690 ERROR akka.cluster.sharding.DDataShardCoordinator - The ShardCoordinator was unable to update a distributed state within 'updating-state-timeout': 5000 millis (retrying). Perhaps the ShardRegion has not started on all active nodes yet? event=ShardRegionRegistered(Actor[akka://cloudstate-proxy@172.17.0.9:2552/system/sharding/eventsourced-shopping-cart#1207442416])
2021-02-02 00:05:09.125 WARN akka.actor.ActorSystemImpl - Failed to load JFR flight recorder, falling back to noop. Exception: java.lang.ClassNotFoundException: akka.cluster.sharding.internal.jfr.JFRShardingFlightRecorder
2021-02-02 00:05:09.130 INFO com.datastax.driver.core.GuavaCompatibility - Detected Guava >= 19 in the classpath, using modern compatibility layer
2021-02-02 00:05:09.139 INFO com.datastax.driver.core - DataStax Java driver 3.7.2 for Apache Cassandra
2021-02-02 00:05:09.140 INFO com.datastax.driver.core.Native - Could not load JNR C Library, native system calls through this library will not be available (set this logger level to DEBUG to see the full stack trace).
2021-02-02 00:05:09.140 INFO com.datastax.driver.core.ClockFactory - Using java.lang.System clock to generate timestamps.
2021-02-02 00:05:09.140 INFO com.datastax.driver.core.ClockFactory - Using java.lang.System clock to generate timestamps.
2021-02-02 00:05:09.142 INFO com.datastax.driver.core.NettyUtil - Did not find Netty's native epoll transport in the classpath, defaulting to NIO.
2021-02-02 00:05:09.143 INFO com.datastax.driver.core.Native - Could not load JNR POSIX Library, native system calls through this library will not be available (set this logger level to DEBUG to see the full stack trace).
2021-02-02 00:05:09.144 INFO com.datastax.driver.core.utils.UUIDs - PID obtained through JMX: 1
2021-02-02 00:05:09.144 INFO com.datastax.driver.core.ClockFactory - Using java.lang.System clock to generate timestamps.
2021-02-02 00:05:09.369 INFO com.datastax.driver.core.policies.DCAwareRoundRobinPolicy - Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
2021-02-02 00:05:09.369 INFO com.datastax.driver.core.Cluster - New Cassandra host cassandra.default.svc.cluster.local/10.107.191.162:9042 added
2021-02-02 00:05:09.409 INFO com.datastax.driver.core.policies.DCAwareRoundRobinPolicy - Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
2021-02-02 00:05:09.409 INFO com.datastax.driver.core.Cluster - New Cassandra host cassandra.default.svc.cluster.local/10.107.191.162:9042 added
2021-02-02 00:05:22.470 INFO com.datastax.driver.core.ClockFactory - Using java.lang.System clock to generate timestamps.
2021-02-02 00:05:22.590 INFO com.datastax.driver.core.ClockFactory - Using java.lang.System clock to generate timestamps.
2021-02-02 00:05:29.149 ERROR io.cloudstate.proxy.eventsourced.EventSourcedEntity - Persistence failure when replaying events for persistenceId [eventsourced-shopping-cart|test1]. Last known sequence number [0]
akka.pattern.CircuitBreaker$$anon$13: Circuit Breaker Timed out.
2021-02-02 00:05:29.150 INFO akka.actor.LocalActorRef - Message [io.cloudstate.proxy.entity.EntityCommand] from Actor[akka://cloudstate-proxy@172.17.0.9:2552/temp/eventsourced-shopping-cart$a] to Actor[akka://cloudstate-proxy/system/sharding/eventsourced-shopping-cart/87/test1/entity#-425630766] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://cloudstate-proxy/system/sharding/eventsourced-shopping-cart/87/test1/entity#-425630766] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2021-02-02 00:04:54.928 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2021-02-02 00:04:54.932 WARN akka.actor.ActorSystemImpl - Failed to load JFR flight recorder, falling back to noop. Exception: java.lang.ClassNotFoundException: akka.remote.artery.jfr.JFRRemotingFlightRecorder
2021-02-02 00:04:54.949 INFO akka.remote.artery.tcp.ArteryTcpTransport - Remoting started with transport [Artery tcp]; listening on address [akka://cloudstate-proxy@172.17.0.9:2552] with UID [-2365786287080537531]
2021-02-02 00:04:54.949 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.9:2552] - Starting up, Akka version [2.6.9] ...
2021-02-02 00:04:54.950 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.9:2552] - Started up successfully
2021-02-02 00:04:54.950 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.9:2552] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining
2021-02-02 00:04:54.951 INFO akka.cluster.sbr.SplitBrainResolver - SBR started. Config: stableAfter: 20000 ms, strategy: KeepMajority, selfUniqueAddress: UniqueAddress(akka://cloudstate-proxy@172.17.0.9:2552,-2365786287080537531), selfDc: default
2021-02-02 00:04:54.966 INFO akka.actor.ActorSystemImpl - Starting Cloudstate Proxy version [0.6.0-5-f0446998] running on [Substrate VM GraalVM 20.3.1 Java 11]
2021-02-02 00:04:54.967 INFO akka.actor.ActorSystemImpl - Registering SIGTERM handler...
2021-02-02 00:04:54.967 INFO akka.management.internal.HealthChecksImpl - Loading readiness checks [(cloudstate,io.cloudstate.proxy.HealthCheckReady)]
2021-02-02 00:04:54.967 INFO akka.management.internal.HealthChecksImpl - Loading liveness checks [(cloudstate,io.cloudstate.proxy.HealthCheckLive)]
2021-02-02 00:04:54.969 INFO akka.management.scaladsl.AkkaManagement - Binding Akka Management (HTTP) endpoint to: 172.17.0.9:8558
2021-02-02 00:04:54.969 INFO akka.management.scaladsl.AkkaManagement - Including HTTP management routes for ClusterBootstrap
2021-02-02 00:04:54.969 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Using self contact point address: http://172.17.0.9:8558
2021-02-02 00:04:54.969 INFO akka.management.scaladsl.AkkaManagement - Including HTTP management routes for HealthCheckRoutes
2021-02-02 00:04:54.975 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Initiating bootstrap procedure using kubernetes-api method...
2021-02-02 00:04:54.975 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Bootstrap using `akka.discovery` method: kubernetes-api
2021-02-02 00:04:54.975 INFO akka.management.scaladsl.AkkaManagement - Bound Akka Management (HTTP) endpoint to: 172.17.0.9:8558
2021-02-02 00:04:54.978 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Locating service members. Using discovery [akka.discovery.kubernetes.KubernetesApiServiceDiscovery], join decider [akka.management.cluster.bootstrap.LowestAddressJoinDecider], scheme [http]
2021-02-02 00:04:54.979 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Looking up [Lookup(shopping-cart-cassandra,None,Some(tcp))]
2021-02-02 00:04:54.979 INFO akka.discovery.kubernetes.KubernetesApiServiceDiscovery - Querying for pods with label selector: [cloudstate.io/stateful-service=shopping-cart-cassandra]. Namespace: [default]. Port: [None]
2021-02-02 00:04:55.001 INFO akka.actor.ActorSystemImpl - Prometheus exporter started on /0.0.0.0:9099
2021-02-02 00:04:55.065 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Located service members based on: [Lookup(shopping-cart-cassandra,None,Some(tcp))]: [ResolvedTarget(172-17-0-8.default.pod.cluster.local,None,Some(/172.17.0.8))], filtered to [172-17-0-8.default.pod.cluster.local:0]
2021-02-02 00:04:55.066 ERROR akka.actor.OneForOneStrategy - UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at io.grpc.Status.asRuntimeException(Status.java:533)
	at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
	at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(Thread.java:834)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:8080
Caused by: java.net.ConnectException: Connection refused
	at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_net_ConnectException_2_0002e_0003cinit_0003e_00028Ljava_lang_String_2_00029V(JNIJavaCallWrappers.java:0)
	at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
	at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
	at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:834)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
2021-02-02 00:04:55.086 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Contact point [akka://cloudstate-proxy@172.17.0.8:2552] returned [1] seed-nodes [akka://cloudstate-proxy@172.17.0.8:2552]
2021-02-02 00:04:55.087 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Joining [akka://cloudstate-proxy@172.17.0.9:2552] to existing cluster [akka://cloudstate-proxy@172.17.0.8:2552]
2021-02-02 00:04:55.087 INFO akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message [akka.management.cluster.bootstrap.internal.BootstrapCoordinator$Protocol$BootstrappingCompleted$] from Actor[akka://cloudstate-proxy/system/bootstrapCoordinator#15500066] to Actor[akka://cloudstate-proxy/deadLetters] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cloudstate-proxy/deadLetters] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2021-02-02 00:04:55.378 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.9:2552] - Received InitJoinAck message from [Actor[akka://cloudstate-proxy@172.17.0.8:2552/system/cluster/core/daemon#-754755589]] to [akka://cloudstate-proxy@172.17.0.9:2552]
2021-02-02 00:04:55.380 INFO akka.cluster.Cluster - Cluster Node [akka://cloudstate-proxy@172.17.0.9:2552] - Welcome from [akka://cloudstate-proxy@172.17.0.8:2552]
2021-02-02 00:04:56.179 ERROR akka.actor.OneForOneStrategy - UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at io.grpc.Status.asRuntimeException(Status.java:533)
	at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
	at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(Thread.java:834)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:8080
Caused by: java.net.ConnectException: Connection refused
	at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_net_ConnectException_2_0002e_0003cinit_0003e_00028Ljava_lang_String_2_00029V(JNIJavaCallWrappers.java:0)
	at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
	at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
	at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:834)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
2021-02-02 00:04:58.583 INFO akka.management.internal.HealthChecksImpl - Check [io.cloudstate.proxy.HealthCheckReady] not ok
2021-02-02 00:04:58.672 INFO io.cloudstate.proxy.EntityDiscoveryManager - Received EntitySpec from user function with info: ServiceInfo(,,OpenJDK Runtime Environment 11.0.10+9,cloudstate-java-support,0.6.0-4-f0446998,0,2,UnknownFieldSet(Map()))
2021-02-02 00:04:58.672 DEBUG io.cloudstate.proxy.eventsourced.EventSourcedSupportFactory - Starting EventSourcedEntity for eventsourced-shopping-cart
2021-02-02 00:04:58.674 INFO akka.cluster.singleton.ClusterSingletonManager - ClusterSingletonManager state change [Start -> Younger]
2021-02-02 00:04:58.674 INFO EventingManager - Eventing support turned off in configuration
2021-02-02 00:04:58.674 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Post(/cart/{user_id}/items/add),*,,Vector(),UnknownFieldSet(Map()))]
2021-02-02 00:04:58.675 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Post(/cart/{user_id}/items/{product_id}/remove),,,Vector(),UnknownFieldSet(Map()))]
2021-02-02 00:04:58.675 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Get(/carts/{user_id}),,,Vector(HttpRule(,Get(/carts/{user_id}/items),,items,Vector(),UnknownFieldSet(Map()))),UnknownFieldSet(Map()))]
2021-02-02 00:04:58.676 DEBUG io.cloudstate.proxy.EntityDiscoveryManager - Starting gRPC proxy
2021-02-02 00:04:58.680 INFO io.cloudstate.proxy.EntityDiscoveryManager - CloudState proxy online at /0.0.0.0:8013
2021-02-02 00:05:09.122 DEBUG io.cloudstate.proxy.HttpApi$HttpEndpoint - Received HTTP request [/carts/test1] for [com.example.shoppingcart.ShoppingCart.GetCart]
  * Request Field: [com.example.shoppingcart.GetShoppingCart.user_id] = [test1]
2021-02-02 00:05:09.122 DEBUG io.cloudstate.proxy.Serve$ - Received gRPC request [/com.example.shoppingcart.ShoppingCart/GetCart]
2021-02-02 00:05:29.134 INFO akka.actor.ActorSystemImpl - Request timeout encountered for request [GET /carts/test1 Empty]

=== User function logs ===

2021-02-02 00:04:56.069 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$AddLineItem
2021-02-02 00:04:56.075 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.google.protobuf.Empty
2021-02-02 00:04:56.077 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$RemoveLineItem
2021-02-02 00:04:56.079 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$GetShoppingCart
2021-02-02 00:04:56.081 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$Cart
[INFO] [02/02/2021 00:04:58.893] [StatefulService-akka.actor.default-dispatcher-6] [akka.actor.ActorSystemImpl(StatefulService)] Received discovery call from [cloudstate-proxy-core 0.6.0-5-f0446998] supporting Cloudstate protocol 0.2
2021-02-02 00:04:47.408 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$AddLineItem
2021-02-02 00:04:47.411 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.google.protobuf.Empty
2021-02-02 00:04:47.413 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$RemoveLineItem
2021-02-02 00:04:47.414 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$GetShoppingCart
2021-02-02 00:04:47.416 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$Cart
[INFO] [02/02/2021 00:04:50.179] [StatefulService-akka.actor.default-dispatcher-6] [akka.actor.ActorSystemImpl(StatefulService)] Received discovery call from [cloudstate-proxy-core 0.6.0-5-f0446998] supporting Cloudstate protocol 0.2
[ERROR] [02/02/2021 00:05:29.161] [StatefulService-akka.actor.default-dispatcher-10] [EventSourcedImpl] Terminating entity due to unexpected failure
io.cloudstate.javasupport.impl.EntityExceptions$EntityException: Protocol error: Expected Init message
	at io.cloudstate.javasupport.impl.EntityExceptions$ProtocolException$.apply(EntityExceptions.scala:46)
	at io.cloudstate.javasupport.impl.eventsourced.EventSourcedImpl.$anonfun$handle$1(EventSourcedImpl.scala:125)
	at akka.stream.impl.fusing.Map$$anon$1.onPush(Ops.scala:53)
	at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:541)
	at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:527)
	at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:390)
	at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:625)
	at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:502)
	at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:600)
	at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:769)
	at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$shortCircuitBatch(ActorGraphInterpreter.scala:759)
	at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:785)
	at akka.actor.Actor.aroundReceive(Actor.scala:537)
	at akka.actor.Actor.aroundReceive$(Actor.scala:535)
	at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:691)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:577)
	at akka.actor.ActorCell.invoke(ActorCell.scala:547)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
	at akka.dispatch.Mailbox.run(Mailbox.scala:231)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)

2021-02-02 00:04:55.907 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$AddLineItem
2021-02-02 00:04:55.911 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.google.protobuf.Empty
2021-02-02 00:04:55.913 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$RemoveLineItem
2021-02-02 00:04:55.915 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$GetShoppingCart
2021-02-02 00:04:55.916 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$Cart
[INFO] [02/02/2021 00:04:58.609] [StatefulService-akka.actor.default-dispatcher-5] [akka.actor.ActorSystemImpl(StatefulService)] Received discovery call from [cloudstate-proxy-core 0.6.0-5-f0446998] supporting Cloudstate protocol 0.2

Exited with code exit status 1
CircleCI received exit code 1
@pvlugter pvlugter mentioned this issue Feb 2, 2021
Merged
@sleipnir
Copy link
Contributor

sleipnir commented Feb 2, 2021

Judging by these logs it seems to me that the proxy is not activating the entity first.

@pvlugter
Copy link
Member Author

pvlugter commented Feb 2, 2021

Yeah, it's not managing to initialise the entity first. I think the timeout on replaying events will be the issue:

ERROR io.cloudstate.proxy.eventsourced.EventSourcedEntity - Persistence failure when replaying events for persistenceId [eventsourced-shopping-cart|test1]. Last known sequence number [0]
akka.pattern.CircuitBreaker$$anon$13: Circuit Breaker Timed out.

The user function error, that it expected an init message, is not as useful without saying what it received instead.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pvlugter @sleipnir