Upgrades

Upgrading is one operation: Platform N → N+1, for your cluster’s exact recorded profile set.

Not eight component upgrades you sequence yourself and hope about. One transition, tested against your profile set before it reached you, with gates in front of it and a way back.

This page is long because this is the operation people are afraid of, and because you should be able to run it, understand it, and recover from it without talking to us.

What an upgrade actually changes

The bundle version determines every component version. Moving from Platform N to N+1 may change any of them, including the Kubernetes version itself. What changes between two specific bundles is in the bundle manifest, and you can see the diff before you start:

kubenest platform diff --from 0.9 --to 1.0

Your profile set does not change during an upgrade. Adding or removing a profile is a separate operation.

Pre-flight gates

Seven gates run before anything is touched. Any gate that fails stops the upgrade with nothing changed.

These are not ceremony. Each one exists because skipping it produces a specific, known bad outcome. All seven run even after one has failed: being told to fix one thing, re-running and hitting the next is the tool failing you rather than your cluster.

The fresh backup most people expect to find in that table is not a gate: it is stage 2, the first thing that runs once the gates pass. A failure there still leaves nothing changed, and it is still important information in its own right — see when a stage fails.

The deprecated API scan

This is the gate that separates an upgrade product from a cron job, and it is the one worth understanding.

Kubernetes removes APIs. When a minor version drops an API group your workloads use, the upgrade succeeds, the cluster comes up healthy, every platform component reports Ready — and your application stops working, because the manifests it deploys from now reference an API that no longer exists.

An upgrade that cleanly upgrades the cluster and takes down your product has actively harmed you. It is worse than no upgrade at all.

So before anything else, KubeNest scans your live workloads for API versions removed or deprecated in the target Kubernetes version. If it finds any, the upgrade is blocked and the report names the resource, its namespace, the API version it uses, and the version it should move to.

The scan covers your Git desired state as well as the live cluster, and the two are treated differently. A live object using a removed API breaks during the upgrade, so it blocks. A manifest in Git that is not applied yet breaks at the next ArgoCD sync — soon, but not during the upgrade — so it warns, loudly, naming the file and the ref. The distinction is deliberate: a live finding is our upgrade breaking your product, while a Git finding is your repository about to. If Git findings blocked, a stale branch nobody has touched in a year could hold up a security patch in a repository your platform team may not even own.

Scanning only the live cluster would leave the hole where the upgrade succeeds, ArgoCD syncs minutes later, the removed API comes back from Git, and the customer blames the upgrade — correctly, because we said we had checked.

BLOCKED: 3 resource(s) use APIs removed in Kubernetes v1.36.0.

  namespace/payments   Ingress api-gateway
      networking.k8s.io/v1beta1  →  networking.k8s.io/v1

  namespace/payments   HorizontalPodAutoscaler worker
      autoscaling/v2beta2        →  autoscaling/v2

  namespace/internal   CronJob nightly-report
      batch/v1beta1              →  batch/v1

Fix these, then re-run. Nothing has been changed.

If a resource is genuinely safe — it is inert, or you are removing it in the same
window — accept it by name and only by name:
  --acknowledge payments/Ingress/api-gateway
  --acknowledge payments/HorizontalPodAutoscaler/worker
  --acknowledge internal/CronJob/nightly-report

Fixing them is your work, in your manifests — we cannot rewrite your application for you. Once they are fixed and redeployed, re-run the upgrade.

There is no --force. A blanket override is how a customer takes their own product down and then calls us. A finding you have judged safe is accepted one resource at a time, by namespace/Kind/name, which is tedious enough to prevent reflexive use and specific enough to audit afterwards:

kubenest platform upgrade --cluster prod-1 --to 1.0 \
  --acknowledge payments/Ingress/legacy-gateway

An API that is deprecated but not yet removed in the target version does not block. It is printed as a warning, because an upgrade that refused every deprecation would refuse most real clusters.

Two properties of the scanner matter more than the scan itself:

• The scanner and its dataset are both pinned in the bundle manifest (pluto, under upgrade.deprecation-scanner). A scanner whose deprecation data has drifted reports you clean against removals it has never heard of, and you believe it. That is confidence it has not earned.
• It fails closed. If pluto cannot be fetched, cannot run, or emits output the parser does not recognise, the gate fails. It never reports “no deprecations found” — a scan that silently degrades into a pass is indistinguishable from a clean cluster at the moment it matters most.

One limit to know: the scan reads what is in the cluster — its live objects and its Helm releases. A manifest that exists only in your GitOps repository, unsynced, is not scanned and will not block. It will break you on the deploy after the upgrade instead, which is the same failure detached from its cause.

Maintenance windows

An upgrade never begins outside its cluster’s configured window.

kubenest cluster set-window --cluster prod-1 \
  --days sat,sun --start 02:00 --end 06:00 --timezone Asia/Kolkata

The timezone is an IANA name, never an offset: offsets move twice a year, and a window that silently shifts by an hour is worse than no window at all. An end time earlier than the start means the window crosses midnight.

The rule when a window closes mid-upgrade: no new stage starts, but the stage in progress finishes. Abandoning a half-completed stage to respect a clock leaves the cluster in a worse state than the overrun does. If the window closes with stages remaining, the upgrade pauses and resumes at the next window, and the cluster reports that it is mid-upgrade the whole time.

Pre-flight also compares the estimated duration against the time left in the window — fixed overhead plus a per-node cost, both from the manifest — and refuses to start an upgrade that cannot finish inside it. Starting one you know will not fit is choosing the overrun in advance.

Four stages are exempt from that pause. preflight is where the window is checked in the first place, so pausing before it would be circular. backup takes the snapshot a rollback depends on, which is not skipped for a clock. verify and record must not be separated from the transition they describe — pausing there would leave a cluster upgraded and unverified until the next window. Everything expensive sits between backup and verify, and none of it is exempt.

The order of operations

Components first. Kubernetes last.

This ordering is deliberate and it is the most important design decision on this page: everything before the k3s stage is cheaply reversible, and the k3s stage is the point of no return. Platform components roll back with a Helm rollback in seconds. Kubernetes does not roll back at all — see what actually rolls back. Putting the irreversible step last means the great majority of failures happen while retreating is still cheap.

The diagram is the argument for the ordering. Every stage before 6 exits through the cheap path on the left; only 6 and 7 exit through a datastore restore. Five of the seven ways an upgrade can fail cost you seconds, and that is a consequence of sequencing, not of any recovery machinery.

Every stage converges on its own health check before the next one starts, and a check that fails halts the sequence rather than continuing. Continuing past a failed check is how one broken component becomes a broken cluster.

A component stage is not satisfied by “the pods are Ready”, because the previous version’s pods are Ready too — a chart pin that cannot resolve leaves the old release running and everything green. It requires that the release reports the pinned version and that the job applying it succeeded. That distinction was found by a real cluster, where a deliberately poisoned pin otherwise sailed through.

Stage 6 upgrades nodes one at a time. Each node is cordoned, drained, upgraded, and must return to Ready and pass its health check before the next node is touched. Servers complete before any agent starts: an agent joining a control plane it is newer than is the one version skew Kubernetes does not promise to tolerate.

Every wait in that sequence has a deadline

Each step below takes its limit from limits.timeouts in the bundle manifest, and reports converging with visible progress until the deadline rather than sitting silent:

Halting is not rolling back. An expired deadline stops the sequence and leaves the cluster where it is, exactly as a failed check does — see when a stage fails. Timing out is evidence that something is wrong, and destroying that evidence automatically is how a stall becomes an outage.

What your workloads experience

Honest expectations, by tier and by workload shape. Replica count alone is not an availability guarantee on the single-server tier. During the control-plane outage, a workload stays available only if enough Ready replicas to serve it are already on agent nodes that remain up:

KubeNest does not taint the control-plane node or restrict application workloads to agents. Unless you have added scheduling constraints and verified the resulting placement, assume the last row applies to you.

Two things are true regardless of tier:

• Draining a node restarts the pods on it. A single-replica workload will restart. There is no way around that, and no upgrade tool can avoid it.
• On the single-server tier, the Kubernetes API is unavailable while the control-plane node upgrades. Running pods on other nodes keep running — the kubelet does not need the API server — but during that window nothing schedules, nothing self-heals, no deploy succeeds, and a pod that dies is not replaced. Workloads that were on the control-plane node were already evicted by its drain; any that found no agent node to land on are down until the node returns. The cluster is frozen, briefly and deliberately.

On the ha tier the API stays available throughout, because servers upgrade one at a time and quorum is maintained. That row is reasoning, not measurement: no cluster has yet been installed or upgraded on three machines — see high availability.

What actually rolls back

“Rollback” means two different things depending on which stage failed, and conflating them is how people end up surprised.

One of them happens on its own and the other waits for you. A failure before the point of no return reverts automatically: it is a Helm-level revert, it takes seconds, it is proven, and there is nothing for a human to weigh. A failure after it does not. Recovering from there is a datastore restore, which rolls all cluster state back to the pre-upgrade snapshot and discards anything else that happened on the cluster in between — that is a judgement about your cluster, not a retry policy, so the upgrade stops, names the stage, and waits for you to run the rollback command.

Platform components — genuine rollback

A failure before the k3s stage takes the seven core components back to the bundle they came from. Each is a Helm release, so reverting is rewriting its resource at the previous pinned version and letting the cluster’s own helm-controller take it back: seconds, nothing lost.

This is the path a forced failure took on a real cluster. Traefik was pinned to a version that does not exist; the upgrade failed at platform-components naming Traefik, chose the component revert rather than a datastore restore, put every component back, and the two-replica workload kept both replicas throughout.

Two limits are worth knowing. Stage 4 never needs reverting because it cannot run — a cluster with any component profile installed is refused there rather than upgraded partially, see profiles. And the agent moved at stage 5 is not part of the component revert; it is left at the version the upgrade installed.

Kubernetes — no such thing as a downgrade

Kubernetes does not support downgrading. Neither does k3s. Once the API server has upgraded and the datastore has written data in the new schema, going back means restoring the datastore snapshot taken at stage 2.

The mechanism is the same on both tiers: an etcd snapshot. single-server runs a single-node embedded etcd and ha runs it replicated across three, but the snapshot, the restore and the command do not differ. Both are covered by the same stage-2 snapshot — see backup and restore — but note that on single-server the restore is also a control-plane outage, because there is only one control plane.

That is a genuine restore, not a rollback: the cluster returns to its state at the moment the snapshot was taken, and there is a service interruption while it happens.

The rollback command

kubenest platform rollback --cluster prod-1

It prints which mechanism it will use — and, for a restore, which snapshot and when it was taken — before touching anything. The cheap mechanism then proceeds. The expensive one stops and requires --confirm, rather than prompting: a datastore restore is a service interruption and should not be one keystroke away.

kubenest platform rollback --cluster prod-1 --confirm

When a stage fails

Every failure names the stage, the component, and what to do next. There is no state this can leave you in that is not covered below.

Failure during preflight. Nothing was changed. Fix what the report names and re-run.

Failure during backup. Nothing was changed. This stage failing is important information in its own right — investigate before upgrading, because it means your backups were not working.

Failure during profiles. Today this is not a failure so much as a refusal: a cluster with any component profile installed stops here rather than being upgraded partially, with core already moved. Roll back, or wait for the profile upgrades to ship.

Failure during platform-components or agent. The cluster is on Kubernetes version N with some components at N+1. Two supported exits, and the failure message prints both:

• Roll back to N with kubenest platform rollback. Fast, no data implications.
• Fix and resume by re-running the upgrade. It reads the stage journal and continues from the failed stage rather than repeating completed ones.

Failure during kubernetes. The most serious case, and the one to read before you start.

Nodes upgrade one at a time, so a failure here means some nodes are on the new version and some are on the old. A mixed-version cluster is a supported state — Kubernetes tolerates a version skew between control plane and nodes — so this is not an emergency. The sequence halts, the cluster keeps serving, and you choose:

• Resume once the failing node’s problem is fixed. This is the normal path and usually the right one.
• Restore to N from the stage-2 datastore snapshot, accepting the service interruption.

Failure during verify. The upgrade completed but a post-upgrade check failed. The cluster is on N+1. The report names the failing check; the cluster is not recorded as successfully upgraded until they pass.

The upgrade was interrupted entirely — your machine died, the network dropped. The cluster records that it is mid-upgrade, and re-running the command resumes from the journal. An interrupted upgrade never leaves the cluster in an unknown state, because the journal is written before each stage rather than after.

Verifying an upgrade

Stage 7 runs the first five automatically, in this order, and names the one that failed. The sixth is yours to check.

These are convergence checks with deadlines, on the same three-outcome rule as the install checks — pass, converging, or fail with the last observed state. A post-upgrade cluster is in more motion than a freshly installed one, because every drained pod is rescheduling at once, so sampling once here is even less defensible than it is at install.

kubectl get nodes -o wide

Stage 8 records the new bundle version against the cluster only after all five pass. An upgrade that completed but failed verification leaves the cluster on N+1 and the record on N, which is the honest state and the one you want to see.

Who starts an upgrade

You do. Every upgrade is started by a person running a command, on every tier, for as long as Platform 1.x is current.

Fleet health tells you when a cluster has fallen behind, and the console shows how far — but nothing applies a bundle to your cluster on its own. Automatic upgrades are a promise the compatibility matrix has to earn before we make it, and that matrix has not run yet. Applying a Kubernetes minor to somebody else’s production while nobody is watching, on the strength of testing we have not done, is not a feature.

The one exception we intend to offer is narrower than “automatic upgrades”: security releases applying themselves inside your maintenance window, with everything else still waiting for a person. A release that exists only to carry a patched component is the case where waiting is the bigger risk, and it is also the case where the change is smallest.

Skipping versions

Step through one bundle at a time. The gate refuses a longer hop.

A target more than one bundle ahead of what you run is refused at pre-flight, naming the bundle to step through first. Two bundles are supported at any time — the current one and the one before it — so falling one release behind is normal and supported, and falling two means one extra hop rather than an unsupported cluster.

A security release does not count against that window. If a bundle exists only because a component needed patching, the release before it stays supported: you did not choose to skip it, we chose to issue it, and the support promise should not shrink because we shipped a fix.

The reason to step at all is the deprecated-API scan. It runs against the Kubernetes version pinned by the bundle you name, and only that one. An API removed in 1.1 and irrelevant again by 1.3 still breaks your workloads at 1.1, and a 1.0 → 1.3 jump would never scan for it. Skipping a bundle is skipping the check that makes the upgrade safe, which is why the answer is a gate rather than advice.