CD with Tanka

[fculpo]

Hi,

I started looking into CD territory (Argo and Flux) with Tanka.
Does anyone already have some setup/architecture working that you could share / talk about ?

I remember that Grafana Team is using Flux + some tk export workflow for instance.

Also I could use advice on organising envs as each of these CD tools relies on the Application/Helm (hence some identified directory) concepts that we currently don’t have/use -> we use a monorepo where Environment = Customer/Stage(Staging/Prod) including all its dependencies, with common bits in /lib or /vendor.

We are struggling on this Environment separation since quite some time now, as each approach (all in one or separated by componant but duplicating this applicative env in each of our customers) has advantages and drawbacks (all in one allows tk apply a whole customer whereas separated envs limit blast radius but complicate the tk apply workflow).

There is also the secrets question, how to gitops them efficiently ?
We use SealedSecret right now, which is a noop on CD, but we would like to switch over to sops which is supported on Flux, but not natively on Argo.

So yeah, I would greatly appreciate some insights and/or shared experiences on integrating Tanka in a GitOps CD scenario.

Thanks.

[nabadger]

I've been playing with Flux v2 / ArgoCD lately with Tanka (currently using Flux v1).

Flux / Argo are flexible enough to handle any directory structure you need - since both let you specify the directory you want to apply commands to (so can you can limit blast radius).

If you are going down the approach of wanting to apply manifests directly (like at runtime via tk apply or kubectl apply), I would always try to limit the blast radius :)

If you pre-render manifests and save them to Git, you can run a lot more tests on them (which makes bulk apply-all safer).

Findings with Flux v2

• No custom plugin support so can't really use tk directly. Instead you can pre-render your manifests (save them to Git) and have Flux use those. I ended up generating via tk export and wrapping with a kustomization.yaml that Flux v2 can use natively.
• Seemed like I had to create a few files per app which is a lot of duplication. I was also playing with the image-updater feature, and ended up with about 5 files to representing a single app...not great.
• Bootstrapping Flux v2 is a bit annoying (everything is heavily focused around Kustomize)

Findings with ArgoCD

• Much easier to get started (bootstrapping is simpler, and these days there are less components than Flux v2).
• Custom Plugins support: I followed https://hodovi.cc/blog/gitops-argocd-and-tanka/
• I use a single Application per app (which is a bit simpler than the Flux approach)

Whilst I don't think this is a great idea, with custom plugins on ArgoCD I can pass values to Tanka: https://tanka.dev/jsonnet/injecting-values#top-level-arguments

This allows me to define a single prod environment in Tanka, but pass through something like clusterName which it can use (i.e. we have prod01, prod02). This may not be an issue in a monorepo, but I have lots of repos, and want to limit any infrastructure changes that have to be pushed down into them.

In general this allows us to de-couple tanka apps from the cluster they are deployed to.

Other stuff

• I tried breaking the manifests on purpose to see what would happen using this method, and basically ArgoCD catches it and doesn't progress the deploying (i.e. if tk show fails for whatever reason).
• Careful about jsonnet-bundler if you use it - ArgoCD will need credentials to pull any private repos you have here - in my case I just re-use the same key ssh deploy-key as the app repo.
• ArgoCD has a useful UI
• ArgoCD has a built-in rollback feature
• ArgoCD has its own RBAC (not a fan of this, but at least you can restrict UI/cli options if needed).

[fculpo]

Thanks for this very detailed answer.
How do you manage as code the dynamic ArgoCD configuration, mainly its argocd-secret Secret ?

When I want to reapply the Getting Started yaml, I always get a diff because Argo seems to modify its objects at runtime.

[nabadger]

We went down the route of deploying ArgoCD via terraform (using the Helm chart) as part of our 'cluster' bootstrap setup - so ArgoCD won't try to modify itself.

• Terraform Deploys ArgoCD via Helm
• Helm chart values.yaml define an initial AppProject and Application which points to our argo-bootstrap.git repo. We manage this with Tanka/Jsonnet and it's responsible for defining extra AppProject resources, and all the Application resources (it's the app-of-apps model)
  • Using tanka on this repo is proving to be useful, since we can easily add new apps per environment (without manually specifying the entire CRD).

[Duologic]

Nice description @nabadger, at Grafana Labs we probably will need to do something similar, bootstrapping CD (Flux v1 here) with the same Terraform that sets up the cluster.

[tsnobip]

Hi, thanks for your detailed post @nabadger. We've started experimenting with argoCD and tanka and it works pretty well overall but there's one thing we're not happy with, when working on a tanka project, we got used to use tk diff to evaluate our changes but diffing on a cluster managed by argoCD creates a lot of noise, dozens and dozens of lines of diff mainly about metadata.

Setting injectLabels to false in spec.json improved the situation a bit, so did using --diff-strategy=subset, but we still have a lot of diff like this when no change has been made:

diff -u -N /var/folders/2k/9dsd6lhj3g5gj0zvj6s75g8w0000gn/T/diff3084189381/LIVE-v1.ServiceAccount.argocd.argo-cd-demo-repo-server /var/folders/2k/9dsd6lhj3g5gj0zvj6s7
5g8w0000gn/T/diff3084189381/MERGED-v1.ServiceAccount.argocd.argo-cd-demo-repo-server
--- /var/folders/2k/9dsd6lhj3g5gj0zvj6s75g8w0000gn/T/diff3084189381/LIVE-v1.ServiceAccount.argocd.argo-cd-demo-repo-server      2022-04-21 17:41:27.000000000 +0200
+++ /var/folders/2k/9dsd6lhj3g5gj0zvj6s75g8w0000gn/T/diff3084189381/MERGED-v1.ServiceAccount.argocd.argo-cd-demo-repo-server    2022-04-21 17:41:27.000000000 +0200
@@ -1,6 +1,13 @@
 apiVersion: v1
+automountServiceAccountToken: true
 kind: ServiceAccount
 metadata:
-  labels: {}
+  labels:
+    app.kubernetes.io/component: repo-server
+    app.kubernetes.io/instance: argo-cd
+    app.kubernetes.io/managed-by: Helmraiser
+    app.kubernetes.io/name: argocd-repo-server
+    app.kubernetes.io/part-of: argocd
+    helm.sh/chart: argo-cd-4.5.3
   name: argo-cd-demo-repo-server
   namespace: argocd

How did you solve this issue? Or more broadly, what is your dev workflow?

edit: since then, I've bumped into this issue (#692), maybe this problem is not directly related to argoCD, I don't know anymore!

[tsnobip]

OK I think we managed to fix the issue!

We first had to make sure to use the same version of tanka locally and in the cluster managed by argoCD (which was by default installing an old version, 0.12).

Using argocd app diff we managed to have exactly the behavior we wanted:

KUBECTL_EXTERNAL_DIFF="git diff" TK_ENV=default argocd app diff argocd --local .  --port-forward --port-forward-namespace argocd --loglevel=warn