What is in the bundle

A KubeNest Platform release is a fixed set of components at fixed versions, tested together as one thing. It has a single version number, and that number is what you run, what we test, what we support, and what we upgrade between.

That last point is the whole reason the bundle exists. Without it, an upgrade is N independent component version bumps whose interactions nobody has tried. With it, an upgrade is Platform 0.9 → 1.0: one transition, tested before it reached you.

A bundle is the unit of test, of support and of upgrade. A cluster is at exactly one bundle version plus a recorded profile set, and that number pins every component on it.

The reasoning behind each choice below — why k3s, why Traefik, why one component per slot — is on Why these choices. This page is what a bundle contains.

Core

Always installed. Never optional. Every component here is present on every KubeNest cluster.

Profiles

Core is what every cluster runs. Profiles are the only supported way to vary it — a named, pinned set of components tested against core as a unit. You choose them at install; the set is recorded against your cluster and travels with it through every upgrade.

Profiles rather than switches, because N independent on/off toggles means 2^N configurations and nobody has tested 2^N of anything. Three switches would be eight combinations per release. Three profiles are five:

N+2, not 2^N. Every one of the five goes through install, workload, backup, restore drill and upgrade on every bundle release. That is the recurring cost of this entire strategy, and it is what “tested together” has to mean to be worth saying.

observability

VictoriaMetrics, Grafana and Loki — metrics, dashboards and logs for your workloads, with default dashboards for cluster health, node resources, workload status and ingress traffic. VictoriaMetrics rather than vanilla Prometheus because it is meaningfully lighter for the same query surface, and less weight is the promise. Retention defaults are sized so the stack cannot quietly fill your disks.

Not to be confused with the fleet health telemetry in core: that reports cluster state to your control plane, this is your own observability for your own applications.

secrets

Encrypted secrets in Git, so the GitOps flow works without plaintext credentials in a repository. secrets is sealed-secrets: self-contained, no external dependency, and it composes with the ArgoCD flow already in the product.

Vault is deliberately not offered as a default. It is a heavy operational commitment, and precisely the kind of thing you are paying us to avoid. Clients who already run Vault or a cloud secret manager want external-secrets, which is its own profile rather than a switch inside this one.

The sealing key is the most important thing in your cluster. Lose it and every sealed secret is unrecoverable, which is why it is in the backup set — see backup and restore.

ha

Three control-plane nodes with embedded etcd instead of one. The only profile that is also an install-time tier: it is a topology stage 3 builds, not components stage 11 installs. What each tier does and does not promise is on HA tiers.

Changing a profile later

secrets is add-only in practice. Removing it uninstalls the controller holding the sealing key, and every SealedSecret in your repository becomes permanently undecryptable. To stop using it, convert every sealed secret to another mechanism first.

ha is chosen at install. Not because the datastore is in the way — every tier runs embedded etcd, so moving to ha is joining two more servers rather than converting anything. Choose the tier you want when you install.

Adding a component profile runs through the upgrade machinery rather than beside it: the same pre-flight gates, journal, rollback, maintenance window and stage reporting. Which gates apply follows from what the change does — an additive profile install moves no Kubernetes version, so there is nothing for a deprecated-API scan to find.

Each profile declares its own backup set — the namespaces and labels that must be captured, and anything like a sealing key whose loss is unrecoverable — and the restore drill covers what it declared.

Fleet health telemetry is core; your observability is not

Two different things that are easy to confuse.

Fleet health telemetry ships in core and is not optional. Its job is to make a problem known before someone has to report it. The agent reports node status, control-plane health, certificate expiry as dates rather than booleans, and backup and restore-drill evidence; the control plane evaluates those into per-check verdicts and routes alerts. The bundle version, profile set, HA tier and volume-group ownership are recorded by the installer’s record stage rather than by the heartbeat.

A group that cannot be collected evaluates to unknown, which is ordered above ok rather than folded into it: a fleet view showing forty green and two grey is the truth, and one showing forty-two green is the failure this exists to prevent.

Your observability — metrics, dashboards, logs for your own workloads — is the observability profile. Different audience, different data, different lifetime.

The manifest

The bundle is machine-readable and versioned. It is the record of what a release contains, and it is what the installer, the upgrade path and the compatibility tests all read.

It has three readers. kubenest platform install fetches the manifest from the control plane, takes every component version and every deadline from it, and its last acceptance check compares the manifest’s pins against what is actually on the cluster before reporting success. kubenest platform upgrade reads both the manifest it is leaving and the one it is moving to: the difference between them is the upgrade plan, and a transition the two manifests do not describe — a downgrade, an unoffered tier — is refused at the gate rather than attempted. And the compatibility matrix builds every tested configuration from it.

bundle: "1.0"
 
core:
  k3s:                        <pinned>
  traefik:                    <pinned>
  gateway-api:                <pinned>
  cert-manager:               <pinned>
  openebs-lvm-localpv:        <pinned>
  velero:                     <pinned>
  system-upgrade-controller:  <pinned>
  kured:                      <pinned>
  kubenest-agent:             <pinned>
 
os:
  # Ubuntu LTS only. The tested release list moves with the bundle, not with the docs.
  supported: [<pinned>]
 
ha-tiers: [single-server, ha]
 
limits:
  # Binary units throughout. 1 GiB = 1024³ bytes, which is what /proc/meminfo and
  # statfs actually report. See "Units" below — this is not pedantry, it decides
  # whether preflight passes a correctly-sized machine.
  resources:
    floor:        { cpu: 2, memory: 3.7Gi, disk: 36Gi }
    recommended:  { cpu: 4, memory: 7.4Gi, disk: 92Gi }
    upgrade-headroom: { disk: 10Gi }   # free space required before an upgrade starts
 
  # Every wait has a limit. Nothing in the platform waits forever.
  timeouts:
    node-ready:        5m    # k3s started, or a node returning after reboot
    component-ready:  10m    # one Helm release reaching its ready condition
    node-drain:       15m    # cordon + drain before reboot or node upgrade
    node-reboot:      20m    # node leaves, comes back, reports Ready
    install-total:    30m    # all 13 stages; the 15-minute budget is the target
    upgrade-per-node: 30m
    backup:            1h
    restore-drill:     2h
 
health:
  # The thresholds fleet health evaluates against. Here rather than in backend
  # code so a release can retune them without a deploy, and so a missing one is
  # a validation error rather than a default nobody can find.
  certificates: { warning-days: 30, critical-days: 7 }
  reporting:    { missed-reports-before-alert: 3, clock-skew-tolerance: 5m }
  nodes:        { not-ready-grace: 25m }
  backup:       { max-backup-age: 48h, max-restore-drill-age: 336h, ... }
  datastore:    { max-snapshot-age: 3h }
 
upgrade:
  # The deprecation dataset is pinned with the bundle. A scan whose data has
  # gone stale reports confidence it has not earned, which is worse than no scan.
  deprecation-scanner: { tool: pluto, version: <pinned>, dataset: <pinned> }
 
backup:
  object-store-plugin: { provider: aws, version: <pinned> }
  defaults:
    datastore-snapshot: { interval: 1h,  keep: 24 }
    workload-backup:    { interval: 24h, keep: 14 }
    restore-drill:      { interval: 7d }
 
profiles:
  # Each profile pins its own components. One component set per name — a slot that
  # could mean either of two things is a slot the test matrix cannot enumerate.
  observability:      { components: { ... } }
  secrets:            { components: { ... } }   # sealed-secrets
  ha:                 { components: {} }        # a topology, not a component set

Limits are part of the bundle, not constants in the code

Every threshold preflight enforces and every window the platform waits through is declared here, versioned with the release, and changed by shipping a new bundle.

This exists because the alternative is worse in a specific way. A timeout hardcoded in the installer is untestable, undocumented, and identical for a two-node lab and a loaded production cluster. Put it in the manifest and it becomes a property of a tested configuration: the release tests can assert against it, it can be measured, an upgrade can change it deliberately, and a support conversation can start with “which bundle, and what were its limits” rather than a guess.

The deadlines are ceilings, not expectations. Real hardware runs well inside them: a single-server core install takes about 4m14s against a 30-minute install-total, and a 0.9 → 1.0 upgrade across two nodes about 3m11s against a 30-minute per-node allowance. A deadline exists so that nothing waits forever, not to describe how long anything takes.

Units

Binary units, everywhere, stated explicitly. A host sold as “8 GB” delivers 7.57 GiB to /proc/meminfo; one sold as “80 GB” leaves about 74.8 GiB on the root filesystem after the image. Those are measured figures from a real cloud host, not a rounding argument.

So a floor written as 8 GB and compared against MemTotal as though it were GiB fails a machine that meets the specification. The floor is a hard failure rather than a warning, which means the customer’s correctly-sized host is refused with a message telling them to buy a bigger one.

Hence 3.7Gi and 36Gi above rather than 4 GB and 40 GB: the same intent — a 4 GB, 40 GB machine — expressed as what the kernel actually reports on one.

Two rules follow, and they apply to any number in this bundle:

1. Every threshold carries its unit in the manifest, in the binary form the check uses.
2. Vendor-nominal figures are for prose only. Anything a check compares against is written in the units the check reads.

What is recorded against your cluster

The manifest says what a release contains. The cluster record says what you have:

cluster-a: Platform 1.0 + []          # core only — no profile installs yet
           ha-tier: single-server
           volume-group: customer-created

Without this, no upgrade is safe. The upgrade path depends on knowing exactly what is installed, and a cluster you did not install last week is otherwise a guess.

Version numbers

A bundle version is major.minor — 1.0, 1.1, 2.0 — and nothing finer. The manifest carries it and the filename repeats it, and the control plane refuses to load a manifest where the two disagree or where either is not two numbers. A release nobody can name unambiguously is not a release.

A new bundle is cut when k3s cuts a minor, and when a component needs a security patch before then. The platform tracks upstream k3s stable minus one minor, pinned to that minor’s current patch, so the ordinary cadence follows Kubernetes rather than our own calendar. Lagging deliberately by one minor is a policy that can be defended to a customer; drifting is not — falling behind on Kubernetes versions is a documented reason teams leave the alternatives.

A major bump means the upgrade asks something of you. A profile removed, a default changed under you, an API version dropped — anything that needs a decision, a migration, or a change on your side before or after the transition. Everything else is a minor, Kubernetes minors included: a k3s minor is a large change to us and, done properly, no change at all to you.

That makes the number answer the only question anyone asks it: will this cost me a maintenance window, or a project? A scheme where every Kubernetes minor burned a major would be easier to administer and would tell you nothing.

When a CVE lands between releases

A vulnerability in a component we ship does not wait for the Kubernetes calendar. It gets its own release: the next minor, containing the patched pin and nothing else.

• A published assessment within two business days. Which component, whether a KubeNest cluster is actually exposed, and what to do in the meantime. Often the answer is that you are not exposed, and that is worth saying quickly rather than accurately-but-next-month.
• A patched bundle within seven days of an upstream fix existing, for anything critical or high in a core component. Medium and low ride the next scheduled bundle.
• The version still means one thing. A security release consumes a minor number — 1.0 → 1.1 — rather than introducing a third digit. One number identifies what you run, and that rule is worth more than making security releases visually distinct.
• It does not shorten your support window. Two bundles are supported at a time, and a release we issued because we needed to patch something does not push the release before it out of support. You skipped nothing.

The honest part: the seven-day clock is for shipping a patched pin we have tested, not for inventing a fix upstream has not written. When the wait is upstream’s, the assessment says so and names what we are waiting on.

This exists because the alternative is what we criticise. We tell you ingress-nginx’s move to maintenance mode — no features, no fixes, no CVE patches — is a reason to leave it. A vendor who makes that argument and cannot describe their own response has not made an argument.

Agent and control plane

The bundle pins the agent. Your control plane upgrades on its own schedule, so the two versions are not locked together, and something has to say how far apart they may drift.

The control plane accepts agents from the current bundle and the one before it — the same two-release window as bundles themselves. A control-plane change that cannot be made compatible with an in-window agent waits for a bundle that moves the agent pin. An agent outside the window is refused with an error naming the bundle to upgrade to, rather than failing in some subtler way later.

Without a stated window, the first control-plane upgrade quietly breaks every cluster still on an older bundle, which is exactly the class of failure the version number exists to prevent.

Platform 1.0 is the current release, and the one to install. The catalog also carries Platform 0.9, the release 1.0 succeeds. It exists because an upgrade cannot be built or tested against a catalog with one entry in it, and because a transition that changes no Kubernetes version never reaches the point of no return — so it would not test the part that matters. It differs from 1.0 in the k3s pin and the agent pin, and both were real releases of the same k3s line when each was cut. Nothing in it is a placeholder, and there is no reason to install it deliberately.