Clusters

A Cluster in kubenest represents a Kubernetes cluster that the operator has been installed on and that the backend can route deployments to. kubenest is designed to manage many clusters simultaneously from a single control plane — each cluster runs its own operator instance, and the hub multiplexes their connections over WebSocket.

How cluster registration works

Registration is a two-phase process: you create a cluster record in the backend (which reserves a UUID and issues a cluster JWT), and then you install the operator on the actual Kubernetes cluster (which uses that JWT to authenticate with the hub).


Backend                    Hub                     Operator (on cluster)
   │                        │                             │
   │  POST /orgs/{id}/      │                             │
   │       clusters         │                             │
   │◄──────────────────────┤                             │
   │  returns cluster_id    │                             │
   │  + connection_token    │                             │
   │                        │                             │
   │             (operator installed via Helm)            │
   │                        │                             │
   │                        │◄────────────────────────────┤
   │                        │  WS CONNECT + JWT           │
   │                        │                             │
   │◄───────────────────────┤                             │
   │  "cluster connected"   │                             │
   │  event                 │                             │
   │                        │                             │
   │  update cluster.status │                             │
   │  to "connected"        │                             │

The operator never needs a public IP or inbound firewall rules. It initiates a WebSocket connection outbound to the hub, authenticates with the JWT, and maintains a persistent session. All subsequent communication — deploy events, status updates, log streams — flows over this connection.

Connection state machine

A cluster record transitions through the following states:


pending  ← (record created; BYO clusters stay here until the operator connects)
  │
  ├──────────────────────┐
  │                      ▼
  │              provisioning  ← (cloud-provisioned only — VMs coming up)
  │                      │
  │                      ▼
  │              awaiting_operator  ← (cloud-provisioned only — infra ready,
  │                      │             waiting for operator to connect)
  ▼                      ▼
connected  ← (operator has authenticated; hub has an active session)
  │
  ├──► disconnected  (hub lost the WebSocket session; operator may be restarting)
  │         │
  │         ▼
  │      connected  (operator reconnects automatically)
  │
  └──► error  (provisioning failed or operator reported a fatal error)

State	Meaning
`pending`	Cluster record created. For a manually registered (BYO) cluster this is the state it sits in until the operator connects
`provisioning`	Cloud provider is creating the VM or managed cluster
`awaiting_operator`	Infrastructure ready; waiting for operator first connect. Only cloud-provisioned clusters enter this state — it is set by the provisioning job when it finishes
`connected`	Operator has an active WebSocket session with the hub
`disconnected`	Hub lost the session; cluster is temporarily unreachable
`error`	Unrecoverable error; details in the cluster’s `message` field

The operator implements automatic reconnection with exponential backoff. Transient disconnects (network blips, operator restarts) recover automatically without user intervention.

Deployments queued while a cluster is disconnected are buffered. When the operator reconnects, it resumes reconciling any pending StackDeploy CRs. The buffering happens at the Kubernetes controller level — the operator reads its own queue of un-reconciled objects on startup.

Multi-cluster topology

You can register any number of clusters with a single kubenest control plane. Common topologies include:

Environment-per-cluster. Separate clusters for development, staging, and production. Each has its own operator, its own GitOps repo branch, and its own ArgoCD instance. Projects and apps are created independently on each cluster.

Region-per-cluster. Multiple clusters across geographic regions. The backend aggregates metrics and status from all of them into a single dashboard view.

Workload-type-per-cluster. A cluster tuned for stateless web workloads (many small nodes) alongside a cluster tuned for stateful data workloads (large storage-optimized nodes). Use per-component clusterID targeting to place components on the right cluster.

Per-component cluster targeting

By default, all components of an App deploy to the cluster associated with the App’s project. For cross-cluster deployments — a web tier on one cluster, a data processing component on another — set cluster_id on the component:

cross-cluster component


{
  "name": "data-processor",
  "type": "workload",
  "cluster_id": "f6g7h8i9-...",
  "workload_spec": {
    "image": "myorg/data-processor:latest",
    "replicas": 4,
    "port": 9000
  }
}

The operator on the target cluster handles this component; the operator on the primary cluster handles the rest. Cross-cluster exportRef is also supported — a workload on cluster A can consume an export from an addon on cluster B, as long as both clusters are connected.

Cross-cluster deployments require both clusters to be connected at deploy time. If the secondary cluster is disconnected, the component targeting it will remain pending until the connection is restored.

Cluster metrics

The backend aggregates CPU and memory capacity and utilization from each connected cluster, updated as the operator sends heartbeat events:

cluster metrics


curl -H "Authorization: Bearer $TOKEN" \
  https://api.your-domain.com/api/v1/clusters/$CLUSTER_ID | jq '{
    cpu_total: .cpu_total,
    cpu_used: .cpu_used,
    mem_total: .mem_total,
    mem_used: .mem_used,
    node_count: .node_count
  }'

These are aggregate cluster-level figures, not per-workload measurements. For per-workload metrics, kubenest supports forwarding Prometheus remote-write data to an external monitoring backend (configured via components.monitoring in the cluster’s Helm values).

Cloud-provisioned clusters

In addition to manually registered (BYOC) clusters, kubenest can provision clusters on supported cloud providers by creating VMs and installing Kubernetes automatically. Provisioned clusters go through the provisioning state and require a CloudCredential record with provider credentials.

Provisioned clusters support scaling:

scale cluster


curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"desired_node_count": 5}' \
  https://api.your-domain.com/api/v1/clusters/$CLUSTER_ID/scale

BYOC (bring-your-own-cluster) registrations are not affected by the scale endpoint — node management for those clusters is handled outside kubenest.

Deleting a cluster

Deleting a cluster record from the backend does not automatically uninstall the operator or clean up the Kubernetes cluster. You must separately:

Uninstall the operator: helm uninstall kubenest-operator -n kubenest-system
Optionally delete the namespace: kubectl delete ns kubenest-system
Delete the cluster record: DELETE /api/v1/clusters/{id}

Deleting a cluster that has active projects will fail unless you delete those projects first.