GitOps and Drift Detection

kubenest uses GitOps as the exclusive mechanism for applying Kubernetes resources. The operator never runs kubectl apply directly. Instead, it writes rendered Helm values to a Git repository and relies on ArgoCD to reconcile the cluster state from those files. This page explains the full path from an API call to running pods, how drift is detected when someone bypasses that path, and how rollbacks work at the Git level.

Why GitOps?

The decision to route all state changes through Git has several practical consequences that are worth understanding before diving into mechanics.

Auditability without instrumentation. Every state change — deploy, patch, rollback, redeploy — produces a Git commit with a timestamp and a structured commit message. You can git log a cluster’s GitOps repository and see a complete history of what was deployed, when, and in what order. No additional audit logging infrastructure is needed.

ArgoCD as an independent enforcement layer. If someone modifies a Kubernetes resource directly (bypassing kubenest entirely), ArgoCD notices the divergence between Git and live state on its next sync cycle and emits a drift event. kubenest surfaces this to the user before the divergence causes a deployment to behave unexpectedly.

Operator-independent recovery. If the kubenest operator goes offline, the cluster continues serving applications as-is. When the operator comes back, it resumes reconciling from the CRD queue. Nothing is lost. ArgoCD continued enforcing the last-known-good state from Git throughout the outage.

Human-readable state. The GitOps repository contains plain Helm values files. A team member who has never used kubenest can read the repository and understand what is deployed. This matters during incidents when you need to move fast.

How a workload reaches Kubernetes

The path from POST /api/v1/apps to running pods involves five distinct stages. Each stage is a discrete unit of work; a failure in any stage stops the process and surfaces an error message that explains which stage failed and why.

Command event dispatch

When you POST an App create (or PATCH an existing App), the backend:

Validates the request body and resolves any exportRef references that can be checked statically.
Records a Deployment row in PostgreSQL with status: pending.
Sends a workload_deploy (or workload_update) command event to the hub, addressed to the target cluster’s operator session.
Returns the new App record immediately with phase: pending.

The backend does not wait for the operator to acknowledge the command. It will learn about success or failure through subsequent status events.

Operator reconcile

The operator’s StackDeploy controller picks up the event and begins reconciling. For a new App, the reconcile loop proceeds component by component in depends_on order. For each component:

Workload components: the Workload controller renders the component’s Helm values from the workload spec (image, replicas, port, env, ingress settings).
Addon components: the Addon controller renders values from the addon spec plus any user-supplied values overrides.
BuildRequest components: if the build mode is dockerfile or buildpack, the BuildRequest controller first submits a build job and waits for the image digest before continuing.

exportRef resolution happens here. If a component’s env var references another component’s export, the operator waits until that component has published its exports before rendering the dependent component’s values. This is why depends_on matters: without it, the operator might attempt to render an env var that does not yet have a value.

Git commit

For each component, the operator writes the rendered values to the GitOps repository at a deterministic path:


clusters/{cluster-id}/namespaces/{namespace}/apps/{app-name}/{component-name}/values.yaml

Each write is a separate Git commit with a structured message:


kubenest: deploy app={app-name} component={component-name} ns={namespace}

The operator uses a dedicated Git identity (kubenest-operator@<cluster-id>) so commits from the operator are visually distinct from any human commits in the same repository.

If two components are being updated concurrently (no depends_on relationship), the operator serializes the commits to avoid merge conflicts — it holds a cluster-scoped mutex during the write phase.

ArgoCD Application upsert

After committing, the operator creates or updates an ArgoCD Application resource in the argocd namespace. The Application points to the path in the GitOps repo that was just committed:


spec:
  source:
    repoURL: https://github.com/your-org/gitops
    targetRevision: main
    path: clusters/{cluster-id}/namespaces/{namespace}/apps/{app-name}/{component-name}
    helm:
      valueFiles:
        - values.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: {namespace}
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

selfHeal: true is what makes ArgoCD the enforcement layer for drift recovery — if live state diverges from Git, ArgoCD re-applies the Git state automatically.

ArgoCD sync and status emission

ArgoCD detects the new commit (or Application update), syncs the cluster state, and updates the Application’s sync.status and health.status fields. The operator watches these fields via a informer on ArgoCD Application resources and emits a status event through the hub whenever they change:


{
  "type": "app.status",
  "cluster_id": "a1b2c3d4-...",
  "namespace": "my-project",
  "name": "my-app",
  "phase": "running",
  "component_statuses": [
    { "name": "web", "phase": "running", "argocd_health": "Healthy", "argocd_sync": "Synced" }
  ]
}

The hub routes this to the backend, which updates the Deployment row and fans the event out as SSE to any connected UI clients.

The GitOps repository layout

A GitOps repository managed by kubenest has this structure:


gitops-repo/
└── clusters/
    └── {cluster-id}/
        └── namespaces/
            └── {project-namespace}/
                └── apps/
                    └── {app-name}/
                        ├── {component-1}/
                        │   └── values.yaml
                        └── {component-2}/
                            └── values.yaml

Each values.yaml is a self-contained Helm values file for that component’s chart. An App with a web workload and a postgres addon will have two files, each corresponding to a separate ArgoCD Application resource and a separate Helm release in the cluster.

This layout means you can understand the state of an entire cluster by reading the filesystem tree. It also means git diff between two revisions shows exactly what changed between two deployments.

Drift detection

Drift occurs when the live Kubernetes state diverges from what the GitOps repository (and therefore the StackDeploy spec) says it should be. The most common causes are manual kubectl edits, direct Helm releases targeting the same resources, or cluster-level autoscalers modifying replica counts.

How drift is detected

ArgoCD detects drift continuously. When it finds a live resource that differs from the Git-committed manifest, it sets the Application’s sync.status to OutOfSync and records the diff. The operator’s ArgoCD Application informer picks this up and emits a drift.detected status event.

The backend stores drift state on the Deployment row and makes it available on GET /api/v1/apps/{ns}/{name}:


{
  "name": "my-app",
  "phase": "running",
  "drift": {
    "detected": true,
    "drift_class": "recoverable",
    "drift_details": [
      {
        "resource": "Deployment/my-app-web",
        "field": "spec.replicas",
        "desired": 2,
        "observed": 1,
        "reason": "Manual kubectl scale"
      }
    ]
  }
}

Drift classes

kubenest categorizes drift into two classes based on the impact on subsequent operations.

recoverable — ArgoCD will self-heal this drift on the next sync cycle (because selfHeal: true is set on the Application). The live resource will be returned to the desired state without any user action. This is the normal class for manual replica count changes or label mutations. The App continues operating normally; the drift is informational.

blocked_sync — The drift conflicts with a pending or in-progress API write. For example, if you are in the middle of a PATCH that changes a component’s image, and someone has simultaneously edited the ArgoCD Application directly to point to a different chart path, the operator cannot safely reconcile both changes. In blocked_sync state:

The API returns 409 Conflict on any write attempt to this App.
The UI shows a drift warning banner with details.
Resolution requires manually reconciling the Git state and then calling POST /api/v1/apps/{ns}/{name}/redeploy to re-assert the desired spec.

blocked_sync is uncommon in practice. It occurs when someone modifies the ArgoCD Application or the GitOps repository contents directly, bypassing kubenest entirely. The safest policy is to treat the GitOps repo as operator-owned and never edit it by hand. If you need to make a one-off change, use the kubenest API instead.

Triggering a redeploy

A redeploy forces ArgoCD to hard-refresh and re-sync a component without changing the spec. It is useful when a pod is stuck in a crash loop due to a transient condition, or when you need to force a rolling restart after rotating a secret.


curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  https://api.your-domain.com/api/v1/apps/my-project/my-app/redeploy

Under the hood, the backend increments the kubenest.io/redeploy-at annotation on the StackDeploy CR. The operator detects the annotation change, triggers an ArgoCD hard-refresh (bypassing the cache), and then forces a re-sync. The result is equivalent to a rolling restart of all workload pods with no spec change.

The redeploy is recorded as a new Deployment row in the history, so you can distinguish a forced redeploy from a natural pod restart.

Rollback mechanics

Every state-changing operation — create, patch, redeploy, rollback — records a Deployment row that includes a prior_state field: a JSON snapshot of the StackDeploy spec as it was immediately before the operation. This snapshot is self-contained: it includes all component specs, depends_on relationships, and resolved parameter values.

Rolling back to a prior deployment replays that snapshot through the same reconcile path:

Identify the target revision

List the deployment history to find the revision you want:


curl -H "Authorization: Bearer $TOKEN" \
  "https://api.your-domain.com/api/v1/apps/my-project/my-app/deployments" | jq '.[] | {id, created_at, phase, message}'

Trigger the rollback


curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"deployment_id": "d4e5f6g7-..."}' \
  https://api.your-domain.com/api/v1/apps/my-project/my-app/rollback

What happens

The backend restores the prior_state spec from the target Deployment row. It then dispatches a workload_update event to the operator with the restored spec. The operator:

Re-renders Helm values from the prior spec.
Commits the prior values to the GitOps repository — producing a new commit (not a git revert) with the old values.
Updates the ArgoCD Applications to point to the new commit.
ArgoCD syncs the cluster state to match the prior values.
For any components that are no longer present in the prior spec (because they were added after the rollback point), the operator deletes the corresponding ArgoCD Applications and removes the Helm releases.

The critical point is that a rollback is a new forward deployment to a prior configuration, not a Git revert and not a Helm rollback. The GitOps history remains linear and intact: you can see exactly when the rollback was triggered and what spec it restored. Running git log on the GitOps repo shows the rollback commit in sequence with all other changes.

This approach also means rollbacks are subject to the same validation as any other deployment — if the prior spec references an image tag that no longer exists in your registry, the rollback will fail with a meaningful error rather than silently deploying a broken configuration.

Addon components within an App are included in the rollback snapshot. If the prior spec had a different chart version for a PostgreSQL addon, rolling back will trigger a Helm downgrade of that chart. Consider the data implications before rolling back an App that contains stateful addons — Helm addon downgrades can be schema-incompatible.