What "Production-Ready" Actually Means for a Kubernetes Cluster

The Problem

A new Kubernetes cluster goes from “demo working” to “production-ready” through a series of capabilities that aren’t visible in any individual feature. The cluster has them all, or it doesn’t. The people responsible for declaring “production-ready” (engineering leadership, security review, a customer’s procurement team) usually don’t have a shared definition of what those capabilities are.

That causes two failure modes:

• A cluster gets declared production-ready and isn’t. The first incident reveals that nothing was set up to recover. The team learns the gap the expensive way.
• A cluster gets blocked from production indefinitely because no two reviewers can agree on what’s missing. The launch slips by quarters.

The fix is a checklist. Not exhaustive. Concrete and scoreable.

The Approach

Five dimensions. Each has eight items. Each item scores 0, 1, or 2:

• 0. Not addressed.
• 1. Partially or informally addressed.
• 2. Fully addressed and operationally verified.

A score of 70+/80 across the dimensions is “production-ready.” 60-69 is “production-soft.” Below 60 is “demo.” The threshold is somewhat arbitrary. The score is what matters: a number leadership can use to compare clusters and track progress.

Dimension 1: Identity & Access

The cluster’s answer to “who can do what.”

1. Cluster-level authentication source is an enterprise identity provider (OIDC, SAML), not static kubeconfigs.
2. RBAC is least-privilege by default. No default-admin bindings. No cluster-admin ClusterRoleBindings to groups larger than a handful of platform engineers.
3. Service accounts are namespaced and tightly scoped. No application uses the default service account.
4. Workload identity (IRSA on EKS, Workload Identity on GKE, AAD Pod Identity on AKS) ties pod-level access to cloud IAM. No long-lived cloud credentials in cluster.
5. Audit logging for the API server is on, shipped to a log-archive store outside the cluster, with at least 90-day retention.
6. MFA enforced on the identity provider for any user with cluster access.
7. Break-glass access is documented: who has it, how it’s used, how it’s audited after.
8. No long-lived kubeconfigs issued to anyone. Access is short-lived and re-authenticated.

Dimension 2: Network

How traffic flows in, out, and between pods.

1. NetworkPolicy enforcement enabled on the cluster (most clusters require an explicit CNI plugin choice for this).
2. Default-deny network policy in every namespace; explicit allow rules for required traffic.
3. Ingress controllers terminate TLS at the edge with managed certificates that auto-renew (cert-manager + ACME is standard).
4. Egress controlled. Either a NAT gateway with logging, an egress proxy, or — for stricter postures — explicit egress NetworkPolicy.
5. Service mesh or sidecar mTLS for service-to-service traffic in regulated environments. Not mandatory for non-regulated workloads.
6. DNS policy doesn’t allow arbitrary external DNS. CoreDNS is configured with explicit forwarders.
7. Load-balancer health checks properly configured (path, interval, threshold) — not the AWS default which often misses application-level health.
8. No services exposed as LoadBalancer or NodePort without explicit review. Ingress is the only sanctioned external entry path.

Dimension 3: Workload Health

What happens when things start to fail.

1. Every deployment has liveness and readiness probes, distinct from each other and not the default “/healthz returns 200” placeholder.
2. Resource requests and limits set on every container. Not optional. Enforced via OPA/Gatekeeper or Kyverno.
3. PodDisruptionBudgets on every production workload, set to allow at most 25% disruption.
4. Horizontal Pod Autoscaler or KEDA configured for workloads with variable load.
5. Anti-affinity rules prevent all replicas of a critical workload from landing on the same node.
6. Quality of Service class is Guaranteed for critical workloads (requests = limits).
7. Init containers have timeouts and are not used for arbitrary one-time setup that should be a Job.
8. Graceful shutdown is implemented in application code. SIGTERM is handled; in-flight requests complete.

Dimension 4: Observability

What the operator can see during normal operation and during incidents.

1. kube-state-metrics scraped by Prometheus or equivalent.
2. Node-level metrics (node-exporter, kubelet) scraped.
3. Application metrics exposed by every service, with at minimum the four Golden Signals (latency, traffic, errors, saturation).
4. Centralized logging ships every container’s stdout/stderr to a queryable store with at least 30-day retention.
5. Audit log retention beyond the centralized logging system (separate retention rule for security events).
6. SLOs defined for every production service, with error budget tracking visible on a shared dashboard.
7. Alerts route to a paging system with documented runbooks. Pages are tested quarterly.
8. Tracing enabled for services in the request path (less critical for backend batch services).

Dimension 5: Recovery

What happens when the cluster, or a piece of it, fails entirely.

1. etcd backups every 30 minutes, retained for 30 days, stored outside the cluster.
2. Application data backed up via Velero or equivalent, with cross-region copies for regulated environments.
3. Restore drill performed at least annually. Documented runbook with timestamps.
4. Documented disaster-recovery posture. RTO and RPO targets stated, mapped to actual backup cadence.
5. HA control plane. Managed services (EKS, GKE, AKS) provide this; self-managed clusters need explicit verification.
6. Multi-AZ workload spread for production deployments via anti-affinity or topology spread constraints.
7. Stateful workloads use StorageClasses with documented backup and snapshot policies.
8. Cluster upgrade procedure documented and rehearsed. Last upgrade was within the support window of the current Kubernetes version.

The Template

Score your cluster row by row. Anywhere you can’t honestly score 2, write a one-line note explaining the gap and the planned remediation.

Total score: __ / 80

• 70-80: Production-ready. Document the score, the date, the cluster, and circulate to whoever needs to approve.
• 60-69: Production-soft. OK for non-customer-facing or non-regulated workloads. Not OK for the production workload your customers depend on.
• Below 60: Demo. Don’t run production traffic. Don’t put PHI on it. Don’t claim it as “production” in customer contracts.

The number you produce will feel low the first time. That’s the point. Most teams score around 50 on the first honest pass. The next quarter’s work is moving 50 to 70.

Operating Notes

Re-score every cluster every six months. Drift happens. Items that scored 2 become 1 because a person left, a process lapsed, a tool changed. The score is a leading indicator.

When a new cluster is built, the scorecard is part of the launch criteria. Cluster cannot host production traffic until the score is documented and approved. This creates an explicit gate that engineering, security, and product can all point at.

When a customer asks “is your platform production-grade?” — share the scorecard methodology, not the raw score. The methodology builds trust; the raw score gets weaponized in contract negotiations.