The Problem
Somewhere around the twentieth service, the deploy pipeline that got you here starts working against you. The pattern is common enough to have a smell: CI builds an image, then runs kubectl apply or helm upgrade against production using cluster credentials stored as pipeline secrets. Push-based deployment. It demos well in month one and decays quietly for the next two years.
The decay has a shape. First, drift: anything applied outside the pipeline (a hot-patched ConfigMap during an incident, a replica bump someone forgot to commit) persists forever, because nothing reconciles the cluster back to a declared state. Second, there is no authoritative record of desired state. The manifests in the repo describe what CI intended at some point in the past; the cluster is the only system that knows what is running now, and the cluster does not do code review. Third, rollback becomes archaeology. At 2 a.m., “roll back” means scrolling CI history, guessing which green build corresponds to the last good state, and re-running it while hoping nobody hand-edited anything since.
For a regulated SMB there is a fourth problem, and it is the one that surfaces in an audit. Your CI system holds credentials that write to production. CI is the most plugin-riddled, most third-party-exposed system you operate, and under this model it has cluster-admin. When the SOC 2 auditor asks how production changes are authorized, reviewed, and logged, the honest answer is “a pipeline job with a shared kubeconfig,” followed by a week of screenshotting build logs. Change control exists as tribal ceremony, not as a property of the system.
The Approach
Two moves, in order. First, invert the write path: nothing pushes to the cluster; a controller inside the cluster pulls from git, and git becomes the only way production changes. Second, make the deployment itself gradual and self-checking, so a bad change is caught by your SLOs and reversed by the machine before a human is paged.
Git as the only write path
GitOps is not “YAML in a repo.” It is a specific inversion: an agent (Argo CD or Flux) runs inside the cluster, watches a git repository, and continuously reconciles the cluster toward what the repo declares. CI never touches the cluster. Its job shrinks to building images and updating an image tag in the deploy repo. Cluster credentials come out of CI entirely; the controller authenticates to git with a read-only deploy key, and the blast radius of a compromised pipeline drops from “production cluster” to “can open a PR.”
The reconciliation loop is the point. It converts drift from slow rot into a visible, correctable event: the controller notices live state diverging from git and either flags it or reverts it. This is the same argument as infrastructure drift detection applied at the workload layer, and it assumes the cluster fundamentals are already in place (production-ready Kubernetes covers that floor). Rollback stops being archaeology: git revert, merge, converge. The last known good state is a commit SHA, not a memory.
A deployment repo that scales
Split app source from desired state. App repos own code, tests, and the Dockerfile; a single deployment repo owns what runs where. The decoupling matters mechanically: CI in an app repo produces an image and opens a commit against the deploy repo bumping a tag. Deploy history and code history stay separate, which is exactly what you want when someone asks what changed in prod this week.
Inside the deploy repo, environments are directories, not branches. Environment branches feel natural and rot without exception. Promotion becomes merging staging into prod, which works until the first hotfix cherry-pick, after which the branches diverge permanently: merge conflicts in generated YAML, prod carrying changes staging never saw, and a history where comparing environments returns noise. With directories, promotion is a PR that copies a change from one overlay to another, and the difference between environments is a diff -r you can run any time.
deploy-repo/
├── apps/
│ └── api/
│ ├── base/ # shared manifests
│ └── overlays/
│ ├── staging/ # 2 replicas, shorter pauses
│ └── prod/ # 6 replicas, full canary
├── platform/ # argo-rollouts, monitoring, ESO
└── argocd/ # ApplicationSet + AppProject definitionsKustomize keeps this honest: base/ holds the manifests, overlays patch replica counts, analysis thresholds, and image tags. Bootstrap with an app-of-apps root Application or an ApplicationSet that generates one Argo CD Application per directory under apps/. Either way, adding a service to the platform is a directory and a PR, not a console session.
Argo CD or Flux, in one paragraph
Argo CD gives you a UI that non-platform engineers use willingly, an RBAC model (AppProjects) that maps cleanly onto team and environment boundaries, and the Argo Rollouts ecosystem for progressive delivery. Flux is lighter, purely CRD-driven, more composable, and marginally easier to operate if nobody ever opens a dashboard. At 10 to 80 engineers the choice matters far less than the six weeks some teams spend debating it. Argo CD wins by default here because the canary machinery below is Argo Rollouts and the UI doubles as the manual sync gate. Pick one, record the decision, stop.
Sync policy, term by term
Argo CD’s syncPolicy has three switches that get cargo-culted together. automated means the controller applies changes as soon as they land in git, no human click. prune means resources deleted from git are deleted from the cluster; without it, removed manifests leave orphans running indefinitely, and an orphaned workload in a HIPAA environment is an unaccounted-for access path. selfHeal means manual edits to live objects are reverted to git state on the next reconcile, usually within minutes. That last one is drift correction in real time, and it will fight anyone who reaches for kubectl edit.
For early adoption: staging gets all three immediately. Prod starts with manual sync for the first month or two: drift and deleted-manifest orphans surface as OutOfSync in the UI (check the prune option when you click sync), but nothing is auto-corrected. The PR review remains the authorization; the sync button in the Argo UI is a timing control, letting you deploy inside a window rather than at merge time. Once the analysis gates below have caught a real regression and on-call trusts them, flip prod to automated, enabling prune and selfHeal at the same time. Manual sync forever is a smell: it means you never built confidence in your gates.
Canary with an SLO gate
Rolling deployments verify that pods start. They say nothing about whether the new version is correct. Argo Rollouts replaces the Deployment with a Rollout resource that shifts traffic in steps and runs an AnalysisRun between steps, querying Prometheus against your actual SLOs.
The sequence that works at this scale: shift 10 percent of traffic to the canary, pause five minutes, run analysis (success rate and p99 latency over the canary pods), then 50 percent, analyze again, then 100. If any AnalysisRun breaches its failureLimit, Rollouts aborts: traffic shifts entirely back to the stable ReplicaSet, which was never scaled down, and the canary scales to zero. Rollback takes seconds and requires no human, no CI run, no image pull. The failed AnalysisRun results and the aborted revision’s pod template stay inspectable for diagnosis instead of being stomped by a redeploy.
Honest constraints. You need real traffic shifting: ALB weighted target groups via ingress annotations, or a mesh; plain Service-based weighting is pod-count-granular and lumpy. And you need enough traffic for the statistics to mean something. At 5 requests per second, a 10 percent canary sees half a request per second: roughly 150 requests over a five-minute window, and about 60 per two-minute query window — workable for success rate, meaningless for p99. Low-traffic internal services should run a simpler two-step canary with longer pauses, or skip canary for blue-green. Do not fake statistical rigor where there is no data.
The audit artifact you get for free
Under this model, every production change is a PR with a named author, a required approver enforced by branch protection, a passing status check, a merge timestamp, and a deterministic link to what the cluster ran (the commit SHA in Argo CD’s sync history). That is change management in the sense auditors mean it, produced as a side effect of shipping. Evidence collection becomes a query over the git log of overlays/prod joined with the Argo CD API, feedable into an evidence pipeline instead of a quarterly screenshot hunt. Keep one-page ADRs in the same repo under docs/adr/ so the reasoning behind canary thresholds and sync policy lives next to the config it explains. Auditors read those too, and it beats reconstructing intent from Slack.
The Template
Deployment-repo layout, a production Rollout with a three-stage canary, and the AnalysisTemplate that gates it. Tune the thresholds to your SLOs, not the other way around.
deploy-repo/
├── bootstrap/
│ └── root-app.yaml # single Application pointing at argocd/
├── argocd/
│ ├── applicationset.yaml # one Application per apps/*/overlays/*
│ └── appproject-prod.yaml # RBAC: who may sync prod
├── platform/
│ ├── argo-rollouts/ # controller install, version pinned
│ ├── external-secrets/ # ESO + ClusterSecretStore (AWS SM)
│ └── monitoring/ # Prometheus, ServiceMonitors
├── apps/
│ └── api/
│ ├── base/
│ │ ├── kustomization.yaml
│ │ ├── rollout.yaml # below
│ │ ├── service-stable.yaml
│ │ ├── service-canary.yaml
│ │ ├── ingress.yaml # ALB, weighted target groups
│ │ └── analysis-template.yaml # below
│ └── overlays/
│ ├── staging/
│ │ └── kustomization.yaml # replicas 2, shorter pauses
│ └── prod/
│ └── kustomization.yaml # replicas 6, image tag pinned here
└── docs/
└── adr/
└── 0007-canary-slo-thresholds.md# apps/api/base/rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: api
spec:
replicas: 6
revisionHistoryLimit: 5
selector:
matchLabels:
app: api
strategy:
canary:
stableService: api-stable
canaryService: api-canary
trafficRouting:
alb:
ingress: api
servicePort: 8080
steps:
- setWeight: 10
- pause: { duration: 5m }
- analysis:
templates:
- templateName: api-slo-check
args:
- name: service
value: api-canary
- setWeight: 50
- pause: { duration: 10m }
- analysis:
templates:
- templateName: api-slo-check
args:
- name: service
value: api-canary
- setWeight: 100
template:
metadata:
labels:
app: api
spec:
containers:
- name: api
# tag is set by CI via `kustomize edit set image` in overlays/prod
image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/api:sha-0000000
ports:
- containerPort: 8080
readinessProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 5
resources:
requests: { cpu: 250m, memory: 256Mi }
limits: { memory: 512Mi }# apps/api/base/analysis-template.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: api-slo-check
spec:
args:
- name: service
metrics:
- name: success-rate
interval: 60s
count: 5
# non-5xx share of requests must stay at or above 99%
successCondition: len(result) > 0 && result[0] >= 0.99
failureLimit: 2 # tolerates two failed measurements; a third aborts
provider:
prometheus:
address: http://prometheus.monitoring.svc.cluster.local:9090
query: |
sum(rate(http_requests_total{service="{{args.service}}",code!~"5.."}[2m]))
/
sum(rate(http_requests_total{service="{{args.service}}"}[2m]))
- name: p99-latency
interval: 60s
count: 5
successCondition: len(result) > 0 && result[0] <= 0.5
failureLimit: 2
provider:
prometheus:
address: http://prometheus.monitoring.svc.cluster.local:9090
query: |
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket{service="{{args.service}}"}[2m])) by (le)
)On abort, Rollouts shifts all traffic to stable and scales the canary down; the revert PR in the deploy repo is the durable record of the rollback. Wire Argo CD Notifications to Slack so an aborted rollout pings the service owner, not just the platform channel.
Operating Notes
Tag bumps are a machine’s job
The commit that changes an image tag in the deploy repo should be opened by CI: a bot PR for prod, a direct commit for staging. Argo CD Image Updater can do this, but a 20-line CI job that runs kustomize edit set image and opens a PR is easier to audit and easier to debug. Humans review promotions; they should not hand-type tags, because hand-typed tags are where typos meet production. The bot PR also gives your prod promotion the same approval trail as every other change, which is the entire premise.
Secrets never enter the deploy repo
A GitOps repo is read widely by design: every engineer, CI, the controller. Kubernetes Secrets, even base64-encoded, do not belong in it. Run External Secrets Operator pointed at AWS Secrets Manager and commit only ExternalSecret references. Secret values then carry their own audit trail in CloudTrail, separate from config history, which is the separation of duties your auditor expects to find anyway. If a secret does land in git, treat it as leaked and rotate; history rewrites are not remediation.
The first month is an argument with selfHeal
Enabling selfHeal surfaces every hand-edit habit your team has, usually within days, usually mid-debugging when someone’s live tweak vanishes under them. This is the system working. Announce the cutover, document the break-glass path (disable selfHeal on a single Application, annotate who and why, restore within 24 hours), and hold the line. Teams that carve out permanent exceptions end up operating two systems: GitOps for the services that behave, and vibes for the ones that page.