# Kaan Systems — Operating Library, full text

> Kaan Systems reviews and builds cloud platforms for funded SaaS and regulated teams: Terraform, CI/CD, Kubernetes, and the audit evidence those systems already produce.

The complete text of all 37 Operating Library guides, newest first. Who Kaan helps, the offers, and how to book are at https://kaansystems.com/llms.txt. The same booking instructions are on https://kaansystems.com/agents. Every article is also available alone as markdown: append `.md` to its URL (e.g. https://kaansystems.com/library/soc2-evidence-map.md). Cite the canonical page URL given at the top of each article, not this file.

Government contracting facts (SAM.gov registration status, teaming, what we do not hold): https://kaansystems.com/government

## Contents

- [Evidence Map: Which SOC 2 Controls Your Terraform Already Proves](https://kaansystems.com/library/soc2-evidence-map) — Platform, updated 2026-10-29
- [From Click-Ops to Code: A Practical Path to Terraform Without Stopping Delivery](https://kaansystems.com/library/click-ops-to-terraform) — Terraform, updated 2026-10-16
- [AI Governance for Regulated Products: A Practical Framework](https://kaansystems.com/library/ai-governance-for-regulated-products) — AI, updated 2026-10-12
- [Which AWS Services Are HIPAA-Eligible? The List and What It Actually Means](https://kaansystems.com/library/hipaa-eligible-aws-services) — Compliance, updated 2026-10-29
- [Do You Need an LLM Gateway? Self-Hosted vs. Managed for Regulated AI](https://kaansystems.com/library/llm-gateway-regulated-ai) — AI, updated 2026-10-12
- [Platform Engineering for Funded SaaS, Health-Tech, and Fintech](https://kaansystems.com/library/platform-engineering-for-regulated-startups) — Platform, updated 2026-10-29
- [Rightsizing AWS Without Breaking Production: A Data-Driven Playbook](https://kaansystems.com/library/rightsizing-aws-without-breaking-production) — FinOps, updated 2026-10-12
- [Savings Plans vs. Reserved Instances vs. On-Demand: The Commitment Decision](https://kaansystems.com/library/savings-plans-vs-reserved-instances) — FinOps, updated 2026-10-12
- [SLOs for Small Teams: Error Budgets Without a Platform Team](https://kaansystems.com/library/slos-for-small-teams) — Platform, updated 2026-10-12
- [SOC 2 for Startups: The Complete Readiness Guide](https://kaansystems.com/library/soc2-for-startups-complete-guide) — Compliance, updated 2026-10-12
- [Vanta vs. Drata vs. Doing It Yourself: SOC 2 Automation Compared](https://kaansystems.com/library/vanta-vs-drata-soc2) — Compliance, updated 2026-10-12
- [Argo CD vs. Flux: Choosing a GitOps Controller for Regulated Kubernetes](https://kaansystems.com/library/argo-cd-vs-flux) — Platform, updated 2026-10-12
- [AWS Control Tower vs. Terraform: The Landing-Zone Build-vs-Buy Call](https://kaansystems.com/library/control-tower-vs-terraform-landing-zone) — Platform, updated 2026-10-12
- [Do You Actually Need an AWS Landing Zone?](https://kaansystems.com/library/when-do-you-need-aws-landing-zone) — Platform, updated 2026-10-12
- [Affordable Zero-Trust Network Access for Growing Teams](https://kaansystems.com/library/affordable-ztna-growing-teams) — Security, updated 2026-10-12
- [Cost-Aware Tagging That Survives Org Changes](https://kaansystems.com/library/cost-aware-tagging) — FinOps, updated 2026-10-12
- [Do You Need a Compliance Platform? Vanta, Drata, and the Honest Answer](https://kaansystems.com/library/do-you-need-a-compliance-platform) — Compliance, updated 2026-10-12
- [Proving Your AI Features Work: Auditability for LLM Systems](https://kaansystems.com/library/llm-feature-auditability) — AI, updated 2026-10-12
- [GitOps and Progressive Delivery with Automated Rollback](https://kaansystems.com/library/gitops-progressive-delivery) — Platform, updated 2026-10-12
- [Killing Long-Lived Credentials with Workload Identity and Dynamic Secrets](https://kaansystems.com/library/killing-long-lived-credentials) — Platform, updated 2026-10-12
- [Building a Multi-Account AWS Landing Zone](https://kaansystems.com/library/multi-account-aws-landing-zone) — Platform, updated 2026-10-12
- [On-Call Without Burning Out the Team](https://kaansystems.com/library/on-call-without-burnout) — Engineering Practice, updated 2026-10-12
- [Vendor Risk Scoring for Regulated SMBs](https://kaansystems.com/library/vendor-risk-scoring) — Security, updated 2026-10-12
- [What Enterprise Buyers Actually Ask About Your AI Features](https://kaansystems.com/library/what-enterprise-buyers-ask-ai) — AI, updated 2026-10-12
- [SOC 2 Readiness in 90 Days: What Engineering Builds](https://kaansystems.com/library/90-day-soc2-readiness-sprint) — Compliance, updated 2026-10-12
- [Adding AI Features Without Breaking Your Compliance Posture](https://kaansystems.com/library/ai-features-compliance-posture) — AI, updated 2026-10-12
- [Zero-Trust for Regulated SMBs Without the Enterprise Price Tag](https://kaansystems.com/library/zero-trust-for-smbs) — Security, updated 2026-10-12
- [The 30-Day Cloud Cost Cleanup](https://kaansystems.com/library/30-day-cloud-cost-cleanup) — FinOps, updated 2026-10-12
- [Choosing Between SOC 2, HITRUST, and HIPAA Attestation](https://kaansystems.com/library/choosing-soc2-hitrust-hipaa) — Compliance, updated 2026-10-12
- [Evidence Pipelines 101: How to Stop Reconstructing Audit Artifacts Quarterly](https://kaansystems.com/library/evidence-pipelines-101) — Compliance, updated 2026-10-12
- [Detecting Infrastructure Drift Without Drowning in Alerts](https://kaansystems.com/library/infrastructure-drift-detection) — Platform, updated 2026-10-12
- [A Minimum-Viable HIPAA Compliance Posture for AWS](https://kaansystems.com/library/minimum-viable-hipaa-aws) — Compliance, updated 2026-10-29
- [Architecture Decision Records: The 1-Page Version](https://kaansystems.com/library/one-page-adrs) — Engineering Practice, updated 2026-10-12
- [A Pragmatic Threat Model for Regulated SMBs](https://kaansystems.com/library/pragmatic-threat-model) — Security, updated 2026-10-12
- [What "Production-Ready" Actually Means for a Kubernetes Cluster](https://kaansystems.com/library/production-ready-kubernetes) — Platform, updated 2026-10-12
- [Reading an Enterprise Security Questionnaire](https://kaansystems.com/library/reading-security-questionnaire) — Sales, updated 2026-10-12
- [Securing PHI in Observability Data](https://kaansystems.com/library/securing-phi-observability) — Compliance, updated 2026-10-12

---
# Evidence Map: Which SOC 2 Controls Your Terraform Already Proves

Canonical: https://kaansystems.com/library/soc2-evidence-map
Topic: Platform · Published: 2026-10-29 · Updated: 2026-10-29
Summary: If your platform runs through Terraform and a gated pipeline, a good share of SOC 2 evidence already exists in your repo. This Evidence Map shows which controls it supports, what artifact to hand over, and where code stops and policy has to take over.

## Key takeaways

- Infrastructure-as-code with a gated pipeline produces evidence as a side effect: change records, policy scans, configuration baselines and drift reports.
- Change management (CC8.1) is the control Terraform supports most directly: every merged pull request records the proposal, review, plan and apply.
- Logical access, boundary protection, encryption and backup controls can be shown from code, provided the code is what's actually running.
- Code can't prove people controls: training, background checks, vendor reviews, risk assessments and incident response still need policy and records.
- Kaan is not an auditor. An independent CPA firm decides what evidence is sufficient; the map shows where your evidence already lives.

**Which SOC 2 controls can your Terraform prove?** If your infrastructure runs through Terraform and a gated CI/CD pipeline, your repo already holds evidence for change management (CC8.1), much of logical access and boundary protection (CC6), configuration monitoring (CC7), and parts of availability (A1). It can't prove the people and policy controls. The Evidence Map below shows which is which.

One note before the map: Kaan Systems is not an auditor. An independent CPA firm audits SOC 2 and decides what evidence is sufficient. What we can do is show you where your evidence already lives, and where it doesn't.

## Why evidence becomes a platform problem

The first SOC 2 audit often turns into a screenshot hunt. Someone opens the console, captures the security group rules, the encryption settings and the IAM policies, pastes them into a folder and hopes nothing changed afterward. Then the auditor asks for a sample of changes from the observation period, and the hunt starts again.

Screenshots show a moment. Auditors care about a period. That's the gap infrastructure-as-code closes: when configuration lives in code and changes go through a pipeline, the record covers the whole period by default.

This isn't a reason to adopt Terraform. Faster environments, safer changes and fewer surprises are. Evidence is the outcome you get for free once the platform is in code. If you're not there yet, start with [from click-ops to code](/library/click-ops-to-terraform).

## The four artifacts that do most of the work

Nearly every row in the map comes back to four artifacts a well-run Terraform setup produces:

1. **The pull request.** Who proposed the change, who reviewed it, the discussion, and the linked ticket if you use one.
2. **The plan and apply logs.** Exactly what changed, when, and by which pipeline identity.
3. **Policy scan results.** Checkov, tfsec or OPA output attached to each pull request, showing what was checked before merge.
4. **Scheduled drift reports.** Proof that what's running still matches the code, between changes.

Branch protection ties them together. If the default branch requires a review and a passing pipeline, and only the pipeline can apply, then the pull request history *is* your change log. We go deeper on capturing and storing these in [evidence pipelines 101](/library/evidence-pipelines-101).

## The Template

### The Evidence Map

One-page SOC 2 Evidence Map for a platform that already runs through Terraform and a gated pipeline. Trace each control to the artifact the repo produces. "Strong for configuration" means the code shows intent; a clean drift report is what shows that intent is running. An auditor decides whether that is sufficient.

| SOC 2 area | What the auditor wants to see | Where it lives in Terraform and CI/CD | Coverage |
|---|---|---|---|
| **CC8.1** Change management | Changes are authorized, tested, approved and implemented in a controlled way | PR history with required review, posted plan, pipeline apply logs, branch protection settings | Strong |
| **CC6.1** Logical access | Access to infrastructure is restricted; encryption protects data | IAM roles and policies in code; KMS or Key Vault keys and encryption settings on storage and databases | Strong for configuration |
| **CC6.2 / CC6.3** Granting and removing access | Access is approved, role-based and removed when no longer needed | Group and role membership in code (if managed there); PR history of grants and removals | Partial: identity provider and HR offboarding records still needed |
| **CC6.6** Boundary protection | Network access from outside is restricted | VPC/VNet design, security groups, firewall rules, private endpoints, WAF in code | Strong |
| **CC6.7** Data in transmission | Data is protected in transit | TLS policies on load balancers, cert-manager or managed certificates, HTTPS-only settings | Strong for configuration |
| **CC6.8** Unauthorized software | Changes to software are controlled | Pinned images and versions, pipeline-only deploys, image scanning in CI | Partial |
| **CC7.1** Detecting configuration changes | Configuration changes and vulnerabilities are detected | Scheduled drift reports, policy-as-code scans, config monitoring enabled in code | Strong |
| **CC7.2** Monitoring | System components are monitored for anomalies | Audit logging (CloudTrail, Activity Log, Cloud Audit Logs), alerting and log retention in code | Partial: someone still has to respond |
| **A1.2** Backup and recovery infrastructure | Backups and recovery infrastructure exist | Backup policies, retention, replication and snapshot schedules in code | Strong for configuration |
| **A1.3** Recovery testing | Recovery plans are tested | Ability to rebuild an environment from code; DR test runbooks and results | Partial: the test itself must happen and be recorded |
| **CC1–CC5, CC9** Governance, risk, people, vendors | Oversight, risk assessment, training, vendor management | Not in Terraform | None: policy and records |

"Strong for configuration" is an important qualifier. Code shows what you *intended*. A clean drift report shows that intent is what's *running*. Auditors will want both, and how much they accept is their decision.

## Tracing one control: CC8.1, end to end

Here's what a single infrastructure change looks like as evidence when the pipeline is set up well:

1. An engineer opens a pull request that tightens a security group rule.
2. CI runs `terraform plan` and posts the diff to the pull request. Policy checks run and pass.
3. A second engineer reviews the plan, not just the code, and approves.
4. Branch protection allows the merge only after the approval and the passing checks.
5. The pipeline applies using a federated, short-lived identity. The apply log records what changed and when.
6. The next scheduled drift check comes back clean, showing production matches the merged code.

When the auditor samples changes from the observation period, each sample is a link to a pull request. No screenshots, no reconstruction from memory.

The weak spot is the exception path. If engineers can still change production from the console, the auditor will ask how those changes are controlled. A documented break-glass process, plus drift detection that catches anything done outside the pipeline, is what closes that gap. Our article on [detecting infrastructure drift without drowning in alerts](/library/infrastructure-drift-detection) covers how to make that check useful rather than noisy.

[Book a call →](https://cal.com/benrubert/30min?utm_source=library&utm_medium=article&utm_campaign=soc2-evidence-map&utm_content=middle)

## Where code stops

Being honest about the limits matters, because overclaiming is how teams get surprised mid-audit. Terraform does not prove:

- That employees completed security training or background checks
- That vendors were reviewed
- That a risk assessment happened and leadership acted on it
- That someone actually responded to an alert, or that an incident was handled according to your plan
- That a quarterly access review happened, unless you export and sign off on it

Some of these can be helped by automation, such as exporting access lists from code every quarter for a reviewer to sign. But the control is the human review, not the export. For the wider picture, see our [SOC 2 for startups readiness guide](/library/soc2-for-startups-complete-guide).

## What to do with the map

Use it to answer three questions about your own environment:

1. **Which rows are "Strong" in theory but not in practice?** For example, IAM is in code, but half the roles were created in the console.
2. **Where is the exception path?** Who can still change production outside the pipeline, and how would you know?
3. **Which rows need a person, not a pipeline?** Assign an owner to each.

The answers also help with enterprise security questionnaires, which ask many of the same questions in different words. [Reading an enterprise security questionnaire](/library/reading-security-questionnaire) shows how to map those questions back to evidence you already have.

## Get your own Evidence Map

Every **Terraform & Platform Assessment** includes a one-page Evidence Map for your environment: which SOC 2 and HIPAA controls your infrastructure-as-code and pipelines already support, and which they don't. It comes alongside a findings report ranked P0/P1/P2, cost quick wins and a 90-day roadmap. It takes 7 business days with read-only access, and is credited 100% toward a retainer signed within 60 days. Book a call to get started.

If you want the evidence to keep flowing after that, our Compliance Evidence add-on keeps policy scans, change records, drift reports and access-review exports flowing into Vanta, Drata, Secureframe or an evidence bucket every month. Your auditor still decides what's sufficient; we make sure the evidence is there when they ask. Book a 15-minute call to talk through fit — we don't publish add-on pricing. [Book the Terraform & Platform Assessment](https://cal.com/benrubert/30min?utm_source=library&utm_medium=article&utm_campaign=soc2-evidence-map&utm_content=close).

**Related reading:** [Evidence pipelines 101](/library/evidence-pipelines-101) · [Detecting infrastructure drift](/library/infrastructure-drift-detection) · [Reading an enterprise security questionnaire](/library/reading-security-questionnaire) · [SOC 2 for startups](/library/soc2-for-startups-complete-guide)

## Frequently asked questions

**Q: Does Terraform make us SOC 2 compliant?**

No. SOC 2 is an attestation issued by an independent CPA firm, and many controls are about people, policy and process. Terraform and a gated pipeline make a large set of technical controls easier to evidence, because the configuration and its change history live in the repo.

**Q: Which SOC 2 control does infrastructure-as-code help with most?**

Change management, CC8.1. When every infrastructure change goes through a pull request with a posted plan, a required review and a pipeline apply, each change has its own record of who proposed it, who approved it and what changed.

**Q: Can an auditor accept a Terraform file as evidence?**

That's the auditor's call. Code shows intended configuration; pair it with proof that the code is what's running, such as a clean scheduled plan or drift report, and with the pipeline history showing how it got there.

**Q: What can't Terraform prove for SOC 2?**

Security awareness training, hiring and background checks, vendor management, risk assessment, board oversight, and the human side of incident response. Those need policies, records and owners.

Template: "One-page SOC 2 Evidence Map (Terraform + CI/CD)" — branded PDF at https://kaansystems.com/templates/soc2-evidence-map.pdf

Source: https://kaansystems.com/library/soc2-evidence-map · Work with Kaan Systems: https://kaansystems.com/contact

---

# From Click-Ops to Code: A Practical Path to Terraform Without Stopping Delivery

Canonical: https://kaansystems.com/library/click-ops-to-terraform
Topic: Terraform · Published: 2026-10-16 · Updated: 2026-10-16
Summary: Most teams don't choose click-ops; it accumulates. This is a five-step path out (inventory, foundation, import in slices, gate every change, watch for drift) that keeps feature work shipping the whole time.

## Key takeaways

- Inventory before you import, and rank the import list by blast radius: networking and identity first, then data stores, then everything else.
- Make Terraform itself safe before importing: remote state with locking, pinned provider and module versions, and no secrets in state or code.
- Import one component at a time with import blocks, and treat a zero-change plan as proof the code matches reality.
- Plan-on-PR with applies from the pipeline is the step that turns Terraform from documentation into control.
- A scheduled drift check closes the loop, so console changes surface in days, not at the next outage.

**How do you move an existing cloud environment into Terraform without stopping delivery?** Inventory what exists, make Terraform itself safe, import one component at a time until each plan shows zero changes, then gate every change behind plan-on-PR and watch for drift. No freeze, no rewrite.

Most software teams don't choose click-ops. It happens. Someone needed a queue on a Friday, an incident needed a firewall rule at 2 a.m., and the first engineer built the database by hand because shipping mattered more. Two years later, part of the platform is in Terraform and part of it only exists in the console.

This post is a practical path out, written for teams with no dedicated platform lead.

## Why it matters now

Click-ops costs show up at the worst moments: when the person who built it leaves, when you need a second environment fast, or when an enterprise customer's security review asks how infrastructure changes are approved. Code answers all three. A screenshot answers none.

There's also a quieter cost. When nobody is sure what's running, every change feels risky, so changes slow down. Engineers stop touching the network because "it works." Upgrades get postponed. The platform starts setting the pace of the product instead of the other way around.

The good news: you don't need a quarter-long migration project to fix it. You need a sequence that lets you make progress in small, safe steps while the rest of the team keeps shipping.

## Step 1: Inventory before you import

List what exists by account or subscription, region and environment. Tag what is already managed by Terraform. Everything else goes on the import list, ranked by blast radius: networking and identity first, then data stores, then everything else.

A few practical notes:

- **Use the cloud's own inventory.** AWS Config or Resource Explorer, Azure Resource Graph and GCP Cloud Asset Inventory will each give you a list faster than clicking through consoles.
- **Compare against state.** `terraform state list` across your existing workspaces tells you what's already managed. The difference between the two lists is your click-ops footprint.
- **Ignore what the cloud manages for you.** Service-linked roles, default resources and auto-generated items usually don't belong in your code. Note them and move on.

If you run more than one AWS account, this is a good moment to look at whether your account structure itself needs work; our guide to [building a multi-account AWS landing zone](/library/multi-account-aws-landing-zone) covers that, and [do you actually need an AWS landing zone?](/library/when-do-you-need-aws-landing-zone) helps you decide whether now is the time.

## Step 2: Fix the foundation first

Before importing anything, make Terraform itself safe:

- **Remote state with locking**, one state per environment and component. One giant state file means every plan touches everything and every lock blocks everyone.
- **Pinned provider and module versions.** An unpinned provider turns a routine apply into a surprise upgrade.
- **No secrets in state or code.** Use your cloud's secret store and reference secrets at runtime. Assume state will be read by more people than you expect.
- **Workload identity for CI.** Your pipeline should authenticate to the cloud with short-lived, federated credentials, not a long-lived access key stored in a CI variable. We cover the pattern in [killing long-lived credentials](/library/killing-long-lived-credentials).

This step is unglamorous and it's where most "partial Terraform" setups fall down. Skip it and every import that follows inherits the problems.

## Step 3: Import in small slices

Use `import` blocks to bring existing resources under management one component at a time. After each import, run a plan and aim for zero changes. A clean plan is your proof the code matches reality.

How to keep slices small:

- **One component per pull request.** "The production VPC and its subnets" is a good slice. "Production" is not.
- **Generate, then clean up.** Recent Terraform versions can generate starter configuration for imported resources. Treat that output as a first draft: rename things, remove defaults you don't need, and move repeated patterns into modules later.
- **Don't refactor while importing.** Get the resource under management exactly as it is. Improve it in a separate change, where the plan shows precisely what will move.
- **Stop on any unexpected diff.** If the plan wants to change or replace something right after import, the code is wrong, not the cloud. Fix the code until the plan is clean.

Delivery keeps going throughout. Each slice is a small, reviewable change that doesn't touch running traffic.

[Book a call →](https://cal.com/benrubert/30min?utm_source=library&utm_medium=article&utm_campaign=click-ops-to-terraform&utm_content=middle)

## Step 4: Gate every change

Add plan-on-PR in CI so every change is reviewed as a diff before it applies. Apply from the pipeline, not from laptops. This is the step that turns Terraform from documentation into control, and it's where change records start creating themselves.

What a good gate looks like:

- **Every pull request posts its plan** as a comment or check, so reviewers see what will actually change, not just what the code says.
- **Applies run from the pipeline** after approval and merge, using the federated identity from Step 2.
- **Console write access shrinks.** Once a component is in code, humans keep read access and a documented break-glass path for emergencies. Anything done through break-glass goes back into code afterward.
- **Policy checks run alongside the plan.** Tools such as Checkov or tfsec catch common misconfigurations before they merge.

Each merged pull request now records who proposed a change, who approved it, what the plan was and when it applied. That's the raw material for answering "how are infrastructure changes approved?" without a single screenshot. If you're curious how far that goes, [evidence pipelines 101](/library/evidence-pipelines-101) walks through it.

## Step 5: Watch for drift

Schedule a drift check (a plan against each environment) and send the result somewhere people read. Drift will happen; the goal is to see it within days, not at the next outage.

The rule for every finding is simple: codify it or revert it. If the console change was right, bring it into code. If it wasn't, let the next apply put things back. Not every difference deserves a page, though. Our guide to [detecting infrastructure drift without drowning in alerts](/library/infrastructure-drift-detection) shows how to tier drift so the dangerous changes get attention and the noise doesn't.

## How long does this take?

It depends on how much lives outside code, how many environments you run, and how much engineering time you can give it each week. A team with one cloud account and a handful of services is in a very different place from a team with several accounts, a Kubernetes cluster and years of manual changes. The inventory in Step 1 is what turns "it depends" into a real plan.

## What you get at the end

New environments in hours rather than weeks, reviewable changes, and a repo that answers "how is this configured and who approved it?" That last one is why we say: audit evidence included.

You also get a platform that more than one person can safely change. That matters most for teams without a senior platform lead, because it means the knowledge lives in the repo instead of in one engineer's memory.

## Where to start

If you're not sure how much of your platform lives outside code, our **Terraform & Platform Assessment** maps it in 7 business days with read-only access: Terraform state and modules, drift, CI/CD, Kubernetes, IAM and network baseline, and cost quick wins, plus a one-page Evidence Map and a 90-day roadmap. Credited 100% toward a retainer signed within 60 days. No Terraform yet? The Assessment becomes a click-ops-to-code review on the same 7-day, read-only scope. [Book the Terraform & Platform Assessment](https://cal.com/benrubert/30min?utm_source=library&utm_medium=article&utm_campaign=click-ops-to-terraform&utm_content=close).

**Related reading:** [Detecting infrastructure drift](/library/infrastructure-drift-detection) · [Building a multi-account AWS landing zone](/library/multi-account-aws-landing-zone) · [Killing long-lived credentials](/library/killing-long-lived-credentials) · [Evidence pipelines 101](/library/evidence-pipelines-101)

## Frequently asked questions

**Q: Do we have to freeze feature work to move to Terraform?**

No. Import one component at a time, aim for a zero-change plan after each import, and leave everything else alone. Delivery keeps going while the managed share of the platform grows.

**Q: Should we rewrite our infrastructure from scratch in Terraform instead of importing it?**

Usually not. Rebuilding means migrating data and traffic, which is where outages happen. Importing what already runs puts it under control without moving it; refactor into cleaner modules later, behind plan-on-PR.

**Q: What should we import first?**

Rank by blast radius. Networking and identity (VPCs or VNets, security groups or firewall rules, IAM roles and policies) come first because a mistake there affects everything, then data stores, then compute and the long tail.

**Q: How do we know the import worked?**

Run a plan right after it. A plan that shows no changes means the code describes what's actually running. Any change in that plan is a mismatch to fix in code before you move on.

Source: https://kaansystems.com/library/click-ops-to-terraform · Work with Kaan Systems: https://kaansystems.com/contact

---

# AI Governance for Regulated Products: A Practical Framework

Canonical: https://kaansystems.com/library/ai-governance-for-regulated-products
Topic: AI · Published: 2026-09-16 · Updated: 2026-10-12
Summary: Every regulated SaaS is bolting an LLM onto its product, and most haven't traced what that does to their compliance boundary. AI governance is the operational discipline that keeps AI features in scope: where the risk lives, the five controls that contain it, the evidence that proves they work, and what enterprise buyers will ask — with links to the deep dive for each.

## Key takeaways

- AI governance for a regulated product is an operating discipline, not an ethics board: the controls, records, and decision gates that keep AI features inside the compliance boundary you already built for SOC 2 or HIPAA.
- An LLM feature opens four new leak points at once — the model vendor as a sub-processor, the PHI/PII you assemble into the prompt, the vendor's retention and training of your data, and the prompt/output logs that become an unplanned regulated-data store.
- The framework is five controls: data-flow mapping, signed BAAs/DPAs with model vendors, retention and no-training terms, human-in-the-loop on consequential output, and model/prompt version tracking.
- Governance is only real if it produces evidence. Every control maps to an artifact — a signed agreement, a data-flow map, a content-free trace, a human-review record, an eval history — that an auditor or buyer can verify.
- AI features fall under your existing SOC 2 and HIPAA umbrella; treat each one as a compliance change, run the same pre-flight before it ships, and re-run it per feature and per vendor, not per company.
- The good-enough bar for a 10-80-engineer regulated SMB is legibility, not sophistication: you can name the vendor, show the agreement, trace the data flow, and prove a human reviews anything consequential.

**What is an AI governance framework, and what does a practical one look like for a regulated product?** An AI governance framework is the set of controls, records, and decision gates that keep your AI features inside the compliance boundary you already built for SOC 2 or HIPAA — data-flow mapping, signed agreements with the model vendors, retention and training controls, human oversight on consequential output, and model/version tracking, each producing evidence an auditor or buyer will accept. It is not an ethics manifesto or a standing AI committee. It is the same operational discipline you apply to any system that touches regulated data, extended to a component that happens to be a probabilistic model behind someone else's API.

AI features do not sit in a separate regime. They fall under the SOC 2 and HIPAA boundary you already run. If that machinery is new, start with the [SOC 2 guide](/library/soc2-for-startups-complete-guide) and the [evidence pipeline](/library/evidence-pipelines-101) those controls should already be writing. The sections below are the AI-specific additions. Governance is what you add so the AI feature doesn't quietly reopen everything that guide had you close.

## Where does AI actually create compliance risk?

An AI feature creates risk in four specific places, and none of them show up in a demo. The demo shows a clean input and a useful output. The audit, the customer security review, and the breach investigation show the four things happening underneath.

- **The LLM provider is a new sub-processor.** The moment your app calls a model API with customer data in the payload, that vendor is processing regulated data on your behalf — the same relationship you have with AWS or your database host, and it carries the same obligations. Vet it the way you'd vet any vendor with standing access to sensitive data: run it through [the vendor risk scoring rubric](/library/vendor-risk-scoring), score the data sensitivity and access depth honestly, and put it on your public sub-processor list. An undisclosed AI sub-processor discovered mid-review is one of the fastest ways to lose a deal.
- **The PHI/PII boundary moves to wherever the prompt goes.** Whatever context you assemble to send the model — the chart note, the record, the message — is now leaving the systems you spent a year putting inside your compliance boundary. If you're on the healthcare path, the boundary you drew in [minimum-viable HIPAA on AWS](/library/minimum-viable-hipaa-aws) just grew an outbound edge, and the minimum-necessary principle applies to prompts as much as to database queries.
- **Retention and training-on-your-data.** Your data doesn't just transit the vendor; it may be retained by them, and on some tiers used to improve their models. This is the risk that turns "we sent one prompt" into "our customer's data is in a training corpus."
- **Prompt and output logging.** Your app logs the prompt for debugging, your observability stack captures it in a trace, your eval pipeline saves inputs and outputs to measure quality — and every one of those is now a regulated-data store you never architected, usually with the wrong retention and the wrong access controls.

The discipline that catches all four before you ship is a pre-flight run per feature. The tactical version — four gates, answered in order, any failure blocks the ship — is [adding AI features without breaking your compliance posture](/library/ai-features-compliance-posture). Governance is what makes that pre-flight a standing control instead of a one-time heroics.

## What goes into an AI governance framework?

Five controls, and each maps to a risk above. This is the whole framework; everything else is elaboration.

1. **Data-flow mapping.** For every AI feature, trace every field that enters the prompt, where it came from, where the response is written, and where any of it is logged. You can't govern a data flow you haven't drawn. The map is also the artifact that answers half of a buyer's questions before they ask.
2. **Signed BAAs / DPAs with model vendors.** No PHI into a prompt without a Business Associate Agreement; no personal data into one without the DPA your privacy commitments require. This is the gate that stops most teams cold because it's the one usually skipped. "We'll get the BAA later" is how regulated data ends up somewhere it can never be recalled from.
3. **Retention and no-training terms.** Confirm in writing that the vendor does not train on your inputs, set the vendor's retention of your data to something you can defend, and re-verify on a schedule because AI vendors change terms far more often than infrastructure vendors do.
4. **Human-in-the-loop on consequential output.** The more consequential the output, the more the human has to be in the decision, not merely notified of it. Draft, don't decide; surface, don't act. This is your best defense against the confidently-wrong answer you didn't catch in testing, and it's the control buyers probe hardest.
5. **Model and prompt version tracking.** Pin dated model snapshots rather than moving aliases, keep prompt templates in version control behind reviewed pull requests, and record which versions were live per request. This is what makes "what was running on March 14" answerable and folds AI behavior into your existing change-management control.

The framework earns its keep only when each control leaves a trace. Governance you can't produce evidence for is a story, and stories fail Type II audits. Map every control to the artifact that proves it:

| Risk area | Control | Evidence |
|---|---|---|
| Model vendor as sub-processor | Vet and score the vendor; sign a BAA/DPA; list on the public sub-processor page | Executed agreement; sub-processor page entry; vendor risk score on file |
| PHI/PII in the prompt | Data-flow map; minimum-necessary field review; de-identify where a BAA isn't possible | Data-flow diagram; field-level review sign-off per feature |
| Training on your inputs | Contractual no-training term; retention set deliberately; quarterly re-verification | Vendor-terms excerpt; "last verified" date; calendar cadence |
| Prompt / output logging | Content-free traces; scoped encrypted store for anything retained; deliberate retention | Trace schema; retention config; a log sample showing shape, not content |
| Model / prompt drift | Pinned dated snapshots; prompt templates in git via reviewed PR; eval on every change | Per-request model id; git log of the template; eval history keyed by version |
| Confidently-wrong output | Human-in-the-loop on consequential output; low-confidence flagging | Human-review records linked to a trace id; a coverage query returning zero unreviewed |
| Model version change | Treat an upgrade as a change with an approval and an eval run | Merged-PR approval; before/after eval scores |

Read the evidence column and notice what it is: the same access, change-management, and logging evidence your SOC 2 or HIPAA program already produces, pointed at a new component. That's the load-bearing idea. You are not standing up a parallel compliance regime for AI — you are extending the one you have.

## How do you prove the framework works?

You prove it with provenance and process, never with stored content — and that distinction is the whole design. When an auditor is in fieldwork with your AI feature in scope, or a buyer's security team runs the diligence call after the questionnaire, the questions converge: what data did the model see, who approved the output before it took effect, what model and prompt version were live on a given date, how do you know quality isn't degrading, and what happened during the incident on X. Every one of those is answerable from metadata.

The mechanism is a content-free trace: one record per request carrying request and tenant ids, timestamp, the pinned model id, the prompt-template version, keyed hashes of each variable input (so you can prove provenance without keeping a second copy of the data), token counts and flags, and the downstream decision — auto-applied, sent to human review with the review id, or discarded. Pair that with human-review records treated as first-class data and a scheduled eval history, and you can answer the diligence call with a query instead of a scramble. The full trace schema, the retention traps, and the reason full prompt logs are the *wrong* answer are in [proving your AI features work: auditability for LLM systems](/library/llm-feature-auditability). The short version: the team that logged everything has eighteen months of PHI-soaked prompts they legally can't hand over, and the team that logged nothing has claims with no artifacts. Content-free traces are the hand that beats both.

The reason the "log the shape, not the content" rule matters twice over is that it's the same rule that keeps request bodies out of your logs generally — the discipline in [securing PHI in observability data](/library/securing-phi-observability), extended one layer up to keep prompts out of your traces.

## What will enterprise and healthcare buyers ask?

They'll ask five clusters of questions, and they aren't testing whether your AI is flawless — they're testing whether you've thought about the AI-specific risks well enough to answer honestly. A CISO doesn't expect your Series B to have a formal AI governance board. They expect you to know where the data goes, who the model vendor is, and what happens when the model is confidently wrong.

The clusters: **data handling** (do you train on our data, where do prompts go, what's the retention), **vendor and sub-processor** (which provider, BAA or DPA in place, are they on your list), **output safety** (can the AI act irreversibly, is there human review, how do you handle hallucinations), **isolation** (is our data isolated at the model layer, could it surface in another tenant's output), and **governance** (who approves AI features, what's the incident plan for a harmful output). The pre-filled answer bank — fifteen of the most common questions with defensible answer patterns — is in [what enterprise buyers actually ask about your AI features](/library/what-enterprise-buyers-ask-ai). Every good answer there is a claim your governance framework has already made true and can prove; the questionnaire is just where the framework becomes visible to the person writing the check.

Notice the loop the cluster closes: the buyer's questions map one-to-one onto the five controls, and the auditability evidence is what makes each answer defensible. Build the framework and the questionnaire answers itself. Skip it and every deal re-litigates the same five themes from scratch.

## The good-enough bar

The bar for a 10-80-engineer regulated SMB is legibility, not sophistication. You do not need an AI ethics board, a governance platform, or a research team red-teaming your prompts. You need to be able to, for any AI feature, name the model vendor and show the signed agreement, produce the data-flow map, demonstrate that prompts are logged as shape and not content, and prove a human reviews anything consequential. If you can do those four things on demand, you clear the bar that keeps you out of a breach notification and through a security review.

The teams that get this wrong aren't reckless. They're moving fast on a feature that looks like every other feature and isn't — one that added a sub-processor, moved a data boundary, and created a log store, all in a single sprint. The framework is cheap precisely because it turns those invisible changes into a visible, answerable checklist, run before the feature ships instead of after the audit finds it.

## The Template

An AI feature governance checklist. Run it before every AI feature that could touch regulated data, and re-run it per feature and per vendor — the BAA you signed for the summarization feature doesn't cover the new vendor you're using for embeddings. Score each item yes / no / not-applicable; any `no` on a required item blocks the ship until it's resolved.

**Data flow (map it first)**
- [ ] Traced every field that enters the prompt; confirmed it's the minimum necessary
- [ ] Confirmed no PHI/PII in URL parameters, feature flags, or analytics events for this feature
- [ ] Traced the response path; confirmed output isn't written anywhere out of scope
- [ ] Data-flow map recorded and attached to the feature

**Vendor + agreements**
- [ ] Model vendor scored on the vendor-risk rubric before access was granted
- [ ] BAA signed with the vendor (or PHI is de-identified before it leaves our systems)
- [ ] DPA in place where personal data is involved
- [ ] Vendor added to the public sub-processor list
- [ ] Data-use terms reviewed: confirmed no training on our inputs, opt-out verified in writing
- [ ] Vendor's retention of our data documented and acceptable

**Logging + retention**
- [ ] Prompts + responses are NOT logged as content to general application logs
- [ ] Traces are content-free (ids, versions, hashes, flags, decision)
- [ ] Anything retained lives in a scoped, encrypted, in-boundary store with production-grade access controls
- [ ] Retention period set deliberately, not left at the vendor default

**Output safety + oversight**
- [ ] Consequential outputs have a human in the decision, not just a notification
- [ ] Low-confidence / flagged outputs are surfaced and logged
- [ ] Users can tell the content was AI-generated
- [ ] A documented plan exists for when the model is confidently wrong

**Versioning + change management**
- [ ] Model version is a pinned dated snapshot, recorded per request (no moving aliases)
- [ ] Prompt templates live in version control and change via reviewed PR
- [ ] A model or prompt change triggers an eval run with a promotion threshold

**Evidence + ongoing**
- [ ] Every control above maps to a produced artifact (agreement, map, trace, review record, eval history)
- [ ] Human-review coverage is queryable: consequential outputs with no linked review = 0 rows
- [ ] This feature is in scope for the next access review + evidence pipeline
- [ ] A re-review triggers when the vendor changes terms or we change the data flow

The checklist takes about an hour per feature. The alternative — an undisclosed sub-processor, an out-of-scope PHI log, and an unanswerable "prove it" during a customer's diligence call — takes a lot longer and usually costs the deal.

## Frequently asked questions

**Q: What is an AI governance framework?**

An AI governance framework is the set of controls, records, and decision gates that keep your AI features inside the compliance boundary you already maintain for SOC 2 or HIPAA. In practice it is five things: mapping the data flow of every AI feature, signing BAAs or DPAs with the model vendors, controlling retention and confirming they don't train on your inputs, putting a human in the loop on consequential output, and tracking model and prompt versions. For a regulated SMB it is not an ethics committee or a written manifesto — it is the same operational discipline you apply to any system touching regulated data, extended to a probabilistic component behind someone else's API, and it only counts when it produces evidence an auditor or buyer can verify.

**Q: Is OpenAI or ChatGPT HIPAA compliant?**

No product is HIPAA compliant on its own — compliance is a property of how you use it, and for a model vendor it turns on a Business Associate Agreement. If protected health information goes into a prompt, the model vendor is processing PHI on your behalf and HIPAA requires a signed BAA, the same as it does for AWS or your database host. The major providers offer BAAs, but not on every tier and not by default: an API customer who requests one and meets the terms can typically sign, while the consumer ChatGPT product does not qualify. Without a BAA you have two honest options — de-identify the data before it leaves your systems, or don't build the feature that way.

**Q: Can LLM vendors train on your data?**

It depends on the product tier and the terms, which is exactly why you have to read them rather than assume. For most enterprise API tiers the answer is no training and opt-out by default, but the default varies by product and the terms change more often than your database host's do. Get the no-training commitment in writing, add the vendor to your sub-processor list, and put a recurring reminder on the calendar to re-verify the terms — a default that quietly flips from 'no training' to 'training unless you opt out' does not send you an email, and it turns a true questionnaire answer into a false one.

**Q: How do you make an AI feature SOC 2 compliant?**

Treat the AI feature as a change subject to your existing controls, not a special case. That means the model vendor is vetted and added to your sub-processor list, prompt templates change through reviewed pull requests so your change-management control covers AI behavior, prompts and responses are logged as shape rather than content so you don't create an out-of-scope data store, model versions are pinned and recorded per request, and consequential outputs have a documented human review. The point is that none of these are new controls — they are your existing access, change-management, and logging controls applied to a new component — and each one has to produce the same kind of evidence the rest of your SOC 2 scope does.

**Q: How do you prove an AI feature works to an auditor?**

With provenance and process, never with stored content. Auditors and buyers converge on a handful of questions — what did the model see, who approved the output, what version was live on a given date, how do you know quality isn't degrading — and every one is answerable from metadata. The mechanism is a content-free trace: one record per request carrying ids, model and prompt versions, keyed hashes of the inputs, flags, and the downstream decision, plus human-review records and a scheduled eval history. Handing over full prompts built from patient records is an impermissible disclosure and the wrong answer; the right answer is a query against traces and review records that returns zero unreviewed consequential outputs.

**Q: What do enterprise buyers ask about AI features?**

The AI section of a security questionnaire clusters into five themes: data handling (do you train on our data, where do prompts go, what's the retention), vendor and sub-processor (which model provider, is there a BAA or DPA, are they on your list), output safety (can the AI take irreversible action, is there human review, how do you handle hallucinations), isolation (is our data isolated at the model layer, could it appear in another tenant's output), and governance (who approves AI features, what's the incident plan for a harmful output). Buyers aren't testing whether your AI is flawless; they're testing whether you've thought about the AI-specific risks well enough to answer honestly in a few specific sentences.

Template: "AI feature governance checklist" — branded PDF at https://kaansystems.com/templates/ai-governance-for-regulated-products.pdf

Source: https://kaansystems.com/library/ai-governance-for-regulated-products · Work with Kaan Systems: https://kaansystems.com/contact

---

# Which AWS Services Are HIPAA-Eligible? The List and What It Actually Means

Canonical: https://kaansystems.com/library/hipaa-eligible-aws-services
Topic: Compliance · Published: 2026-09-16 · Updated: 2026-10-29
Summary: AWS publishes a list of HIPAA-eligible services — the ones covered by its Business Associate Addendum — and healthcare teams routinely misread it. Eligibility is a contractual boundary, not a security rating: it says a service can hold PHI under the BAA, not that yours is configured to. Here is what the list actually means, the common categories, and why you treat it as a runtime check.

## Key takeaways

- "HIPAA-eligible" means a service is covered by the AWS Business Associate Addendum (BAA) — it is the only category of service you may use to store, process, or transmit PHI. Running PHI through a non-eligible service steps outside the BAA entirely.
- Eligibility is not compliance. The list tells you a service can legally hold PHI; it says nothing about whether you have signed the BAA, encrypted the data, scoped access, or turned on logging — all of which are still yours to do.
- The list is long (north of 130 services today and growing) and it changes, so the exact count is the wrong thing to memorize; verify current eligibility against the AWS HIPAA-eligible-services page rather than trusting a one-time decision.
- Core building blocks are almost all eligible — EC2, S3, RDS, DynamoDB, Lambda, EKS/ECS, CloudTrail, KMS — while newer, edge, and generative-AI services are the most likely to lag; some services are eligible only in certain configurations.
- Sign the BAA in AWS Artifact before any PHI lands, and treat eligibility as one control inside a fuller posture, not the whole of it.

**Which AWS services are HIPAA-eligible?** The ones AWS lists as HIPAA-eligible — on the order of 130-plus today and still growing — are the only services covered by the AWS Business Associate Addendum (BAA), which makes them the only services you may use to store, process, or transmit protected health information (PHI). Everything else — a service too new to be added, an edge feature, a region-limited preview — sits outside the BAA, and routing PHI through it is a compliance gap whether or not a single record ever leaks. The list moves, so the count is the wrong thing to memorize; verify each service you touch against the AWS HIPAA-eligible-services page as a runtime check, not a one-time decision.

That is the whole answer, and it is also where most healthcare teams go wrong. They pull the list once, tick the boxes, and treat "eligible" as if it meant "compliant." It does not. Eligibility is a contractual boundary drawn by AWS; compliance is the work you do inside it. This article is about the difference — what the list actually means, which categories it covers, and how to keep from misreading it. It pairs with our [minimum-viable HIPAA posture on AWS](/library/minimum-viable-hipaa-aws), where using only eligible services is control number two of twelve.

## What does "HIPAA-eligible" actually mean?

It means AWS has placed the service in scope of the BAA — nothing more, nothing less. The BAA is the contract that makes AWS your HIPAA *business associate*: it commits AWS to handling PHI according to the Security Rule's requirements on the parts of the stack AWS controls. When AWS declares a service "HIPAA-eligible," it is saying "we will stand behind this service under the BAA." When a service is absent from the list, AWS is making no such commitment, and any PHI you put there is uncovered.

So "eligible" is a legal-coverage status, not a security rating. It does not mean the service is more secure than a non-eligible one. It does not mean the service ships in a compliant configuration. It means one specific thing: this service is inside the contract. Get that framing right and the rest of the list stops being confusing.

Two consequences follow immediately. First, you must actually sign the BAA — it is self-service in AWS Artifact, and until you accept it, *none* of your services are covered, eligible or otherwise. Second, the BAA extends only to the eligible services, so the coverage you just signed for evaporates the moment PHI flows into a service that is not on the list.

## Which AWS services are HIPAA-eligible?

Nearly all the core building blocks are, and the long tail of newer and specialized services is where the gaps live. The table below groups the common categories, names representative services AWS has listed as eligible, and flags the caveat that matters for each. Treat the service names as illustrative and approximate — the authoritative, current list is the AWS HIPAA-eligible-services page, and it changes often enough that you should check rather than trust this snapshot.

| Category | Representative eligible services | Notes / caveats |
|---|---|---|
| Compute | EC2, Lambda, ECS, EKS, Fargate, Batch, Elastic Beanstalk | The core compute plane is well covered. Verify newer runtimes and instance-adjacent features; eligibility attaches to the service, not automatically to every preview capability inside it. |
| Storage | S3, EBS, EFS, FSx, Storage Gateway, AWS Backup | Eligible, but encryption and access controls are entirely on you — an eligible bucket left public is still a breach. Confirm any newer storage tier or feature. |
| Database | RDS, Aurora, DynamoDB, ElastiCache, Redshift, DocumentDB, Neptune | Broadly eligible. Some engines or specific features can carry configuration conditions; check the engine and version you actually run, not just the service name. |
| Networking / delivery | VPC, ELB/ALB/NLB, CloudFront, API Gateway, Route 53, Direct Connect, Transit Gateway | Eligible, but TLS enforcement and private routing are your configuration to get right. Eligibility says nothing about how the endpoint is secured. |
| Analytics | Athena, Glue, EMR, Kinesis, MSK, QuickSight, Data Firehose | Many are eligible; the analytics space adds services quickly, so the newest entrants are the likeliest to lag. Verify before piping PHI-derived data through them. |
| ML / AI | SageMaker, Comprehend Medical, Textract, Transcribe, Rekognition | Established ML services are widely eligible. Generative-AI and the newest AI services are the category most likely *not* yet covered — verify current eligibility explicitly before sending any PHI. |
| Security / logging / mgmt | CloudTrail, CloudWatch, KMS, AWS Config, GuardDuty, Secrets Manager, IAM / Identity Center, Systems Manager | This is your controls layer, and it is well covered. These are the services that make the *rest* of your posture defensible — encryption keys, audit logs, access. |

The pattern is consistent: if it is a foundational, widely deployed service, assume it is eligible and confirm. If it is new, niche, generative-AI-flavored, or a preview, assume it is *not* eligible until you have verified it is. That default — suspicious of the shiny thing, trusting of the boring building block — is the right instinct for regulated infrastructure generally.

## Does HIPAA-eligible mean HIPAA-compliant?

No, and this is the single most expensive misreading of the list. Eligibility gets a service into the room. Compliance is everything you do once it is there.

An eligible service handles PHI compliantly only if you have also: signed the BAA, encrypted the data at rest (KMS) and in transit (TLS 1.2 or higher), scoped access to least privilege, and shipped audit logs somewhere tamper-proof. Skip any of those and you have a non-compliant deployment sitting on a perfectly eligible service. The canonical example is an S3 bucket: S3 is eligible, has been for years, and is on nearly every healthcare team's stack — and a misconfigured, public, unencrypted S3 bucket is one of the most common ways PHI gets exposed. The service being on the list did nothing to prevent that.

This is why we frame eligibility as the *second* control in the [minimum-viable HIPAA posture](/library/minimum-viable-hipaa-aws), not the whole of it. The other eleven controls — account separation, MFA, encryption, tamper-proof logging, backups — are what turn an eligible service into a compliant one. Most of that same control set also carries straight into a [SOC 2 readiness effort](/library/soc2-for-startups-complete-guide), because the encryption, access, and log-integrity requirements overlap almost entirely; if you are still deciding which framework you even owe your buyers, start with [choosing between SOC 2, HITRUST, and HIPAA](/library/choosing-soc2-hitrust-hipaa) before you optimize the service list.

The verdict: eligibility is necessary and nowhere near sufficient. It is the price of entry.

[Book a call →](https://cal.com/benrubert/30min?utm_source=library&utm_medium=article&utm_campaign=hipaa-eligible-aws-services&utm_content=middle)

## Why is eligibility a runtime check, not a one-time decision?

Because the list grows, and your architecture drifts, and the two move independently. AWS adds services to the BAA over time — a service that was not eligible when you evaluated it last year may be eligible now, and, more dangerously, a shiny new service your team just adopted may not be eligible yet even though everything around it is. A one-time "here are our approved services" spreadsheet is stale the day after you write it.

Two failure modes come from treating eligibility as static. The first is an engineer reaching for the newest managed service to solve a problem — a new AI summarization API, a new streaming primitive — without anyone checking whether it is covered, and PHI quietly flowing into it. The second is subtler: PHI ending up somewhere nobody architected for it. Observability is the classic route. Logs, traces, and metrics [capture PHI by accident all the time](/library/securing-phi-observability) — a patient ID in a URL, a full request body in a stack trace — and if that telemetry lands in a tool or service that is not eligible (or outside the BAA entirely, like a third-party SaaS observability vendor without its own BAA), you have a breach that started as a debug log.

The fix is to make eligibility a control you *run*, not a decision you *made*. Practically: keep a living map of every service in your PHI data path, re-check it on a cadence and whenever the architecture changes, and put a gate — a review step, an SCP, a tagging policy — in front of new service adoption in accounts that touch PHI. The AWS HIPAA-eligible-services page is the source of truth; your map is how you know which entries on it you actually depend on.

## What does "eligible only in certain configurations" mean?

It means the service qualifies for the BAA, but only when you use it a specific way — and if you use it another way, you are outside coverage even though the service name is on the list. Eligibility occasionally carries conditions: a particular feature must be enabled or avoided, a specific mode must be used, or certain data must not be sent to a certain surface. AWS documents these conditions alongside the eligible-services list.

The trap is reading the service name, seeing it on the list, and stopping there. "RDS is eligible" is true; "therefore anything I do with RDS is covered" does not follow if a specific feature carries a condition. When a service in your PHI path has configuration caveats, the caveat is part of your control, not a footnote — write it into your service map next to the service, and make sure the way you actually run it matches the way that makes it eligible. When in doubt, read AWS's own conditions for that service rather than inferring from the name.

## So what is the good-enough bar?

A regulated SMB has "gotten eligibility right" when three things are true, and no sooner:

1. **The BAA is signed** in AWS Artifact, before any PHI landed.
2. **Every service in your PHI data path is on the current eligible list** — including the ones PHI reaches by accident (logging, analytics, telemetry), and used in a configuration that keeps it eligible.
3. **Eligibility is a check you re-run**, not a decision in an old doc — with a gate in front of new-service adoption in PHI-touching accounts.

Clear that bar and eligibility is handled. It is a low bar by design, because eligibility is only one control; the encryption, access, and logging that make an eligible service *compliant* are the larger job, covered in the minimum-viable posture. The wrong outcome is the common one: a team that pulled the list once, felt done, and never noticed the day PHI started flowing into a service that was never on it.

Do not over-engineer this. You do not need a bespoke compliance database to track eligibility for a ten-service stack — a maintained page in your runbook and a review gate will do. You do need to stop treating "it's on the list" as the end of the conversation.

## The Template

The service map below is the artifact that keeps eligibility honest. List every AWS service that touches PHI — including the ones PHI reaches indirectly through logs, traces, backups, and analytics — and score each row. Anywhere a row is not clean, decide one of three things: fix it this sprint, accept and document the gap, or escalate it as a compliance risk. Keep it in your runbook and re-run it on a cadence and on every architecture change.

**Prerequisite (do this first)**
- [ ] AWS Business Associate Addendum (BAA) accepted in AWS Artifact
- [ ] A named owner for this map and a review cadence (quarterly, minimum)

**Per-service checklist — one row per service in the PHI data path**
- [ ] Service name and what PHI it touches (stored / processed / transmitted / incidental)
- [ ] Confirmed on the current AWS HIPAA-eligible-services page (date checked: ______)
- [ ] No configuration caveat — or, if there is one, we run it in the eligible configuration
- [ ] Encrypted at rest (KMS) where it holds PHI
- [ ] Enforces TLS 1.2+ in transit where it transmits PHI
- [ ] Access scoped to least privilege
- [ ] Activity logged to the tamper-proof archive

**Don't-forget-these rows (PHI reaches them by accident)**
- [ ] Logging / monitoring destinations (CloudWatch, and any third-party observability tool — does it have its own BAA?)
- [ ] Tracing and APM spans
- [ ] Analytics / data-warehouse pipelines fed from PHI stores
- [ ] Backup and DR targets, including cross-region copies
- [ ] Any queue, cache, or search index in the request path

**Governance**
- [ ] A gate exists (review step, SCP, or tagging policy) before new services are adopted in PHI-touching accounts
- [ ] The map was re-checked against the AWS list on the last review date
- [ ] Every non-clean row has a decision recorded: fix / accept-and-document / escalate

Run it once to find where PHI actually flows, then re-run it whenever the architecture moves. Eligibility is not a fact you learn once; it is a boundary you patrol.

## Frequently asked questions

**Q: Which AWS services are HIPAA-eligible?**

The HIPAA-eligible services are the ones AWS lists as covered by its Business Associate Addendum (BAA) — on the order of 130-plus today and growing. That includes nearly all the core building blocks (EC2, S3, RDS, Aurora, DynamoDB, Lambda, EKS, ECS, VPC, CloudFront, API Gateway, CloudTrail, CloudWatch, KMS, and many more), with newer, edge, and generative-AI services the most likely not yet covered. The list changes over time, and some services are eligible only in certain configurations, so verify current eligibility against the AWS HIPAA-eligible-services page rather than trusting a static list. If a service is not on it, do not use it for PHI.

**Q: What does HIPAA-eligible mean?**

HIPAA-eligible means AWS has placed the service in scope of the AWS Business Associate Addendum (BAA), the contract that makes AWS a HIPAA business associate for that service. It is a contractual boundary, not a security score. Using an eligible service, with the BAA signed, keeps your PHI inside AWS's business-associate coverage; using a service that is not eligible puts that PHI outside the BAA, which is a compliance gap regardless of whether anything ever leaks.

**Q: Does HIPAA-eligible mean HIPAA-compliant?**

No. Eligibility says a service can legally hold PHI under the BAA; it says nothing about whether your deployment is compliant. You still have to sign the BAA in AWS Artifact, encrypt the data at rest and in transit, scope access to least privilege, and log everything to a tamper-proof archive. A perfectly eligible S3 bucket that is public and unencrypted is a breach waiting to happen. Eligibility is the price of entry, not the finish line.

**Q: Is S3 (or Lambda, or DynamoDB) HIPAA-eligible?**

Yes — S3, Lambda, and DynamoDB are all HIPAA-eligible and among the most commonly used services under the AWS BAA, as are EC2, RDS, Aurora, EKS, ECS, and the core networking and security services. Eligibility still assumes you have signed the BAA and configured the service correctly (encryption, access, logging). For any service you are less sure about — especially newer or AI-oriented ones — check the current AWS HIPAA-eligible-services page instead of assuming, because the list moves and some services qualify only in specific configurations.

**Q: Do I need a BAA with AWS?**

Yes. If you store, process, or transmit PHI on AWS you must accept the AWS Business Associate Addendum (BAA), which is available self-service in AWS Artifact. Without it you have no HIPAA coverage for any service, eligible or not. The BAA extends only to AWS's HIPAA-eligible services, so signing it and then keeping PHI on eligible services only are two halves of the same requirement.

**Q: How many AWS services are HIPAA-eligible?**

On the order of 130-plus today, and the number keeps climbing as AWS adds services to the BAA. Because it changes, the exact count is not worth memorizing — what matters is checking whether each specific service in your PHI data path is currently eligible. Treat the count as directional and the per-service check as the real work; verify current eligibility on the AWS HIPAA-eligible-services page.

Template: "HIPAA-eligibility service map" — branded PDF at https://kaansystems.com/templates/hipaa-eligible-aws-services.pdf

Source: https://kaansystems.com/library/hipaa-eligible-aws-services · Work with Kaan Systems: https://kaansystems.com/contact

---

# Do You Need an LLM Gateway? Self-Hosted vs. Managed for Regulated AI

Canonical: https://kaansystems.com/library/llm-gateway-regulated-ai
Topic: AI · Published: 2026-09-16 · Updated: 2026-10-12
Summary: An LLM gateway proxies your model calls so auth, redaction, logging, and key rotation live in one place instead of scattered across every service. For regulated AI the real decision is where that place sits: self-hosted in your VPC keeps prompts and logs in-boundary, a managed gateway offloads the ops but becomes a sub-processor that sees your data, and Amazon Bedrock splits the difference. The comparison, the sub-processor trap, and the good-enough bar.

## Key takeaways

- An LLM gateway is a proxy in front of your model providers that centralizes auth, routing, rate limiting, cost tracking, redaction, logging, and key management. For regulated AI it is the single place you enforce the compliance boundary, not a compliance product you buy.
- The gateway is where four of the five AI-governance controls actually run: scrub PII/PHI before it egresses, restrict traffic to BAA-covered providers, log every call content-free for audit, and rotate provider keys from one place instead of many.
- Self-hosted (LiteLLM, Portkey OSS, and similar) runs in your VPC, so prompts, logs, and keys stay inside your boundary; you trade that for owning the ops. Managed/SaaS offloads the ops but makes the gateway vendor a new sub-processor that sees your prompts.
- Amazon Bedrock is the managed-but-in-your-boundary middle path: AWS runs it, but it sits inside AWS under a BAA that covers its HIPAA-eligible services (verify current eligibility), so a managed experience does not have to mean a new external sub-processor.
- The good-enough bar: for PHI or regulated data, self-host the gateway in-VPC or use Bedrock with a BAA so data and logs stay in-boundary; for de-identified or low-stakes calls a managed gateway is fine, as long as you disclose and score it. Redaction and keys copy-pasted across five services with no chokepoint is the wrong answer at any stakes.

**Do you need an LLM gateway, and does it make your AI compliant?** You need one the moment more than a feature or two sends regulated data to a model provider, and it does not make you compliant on its own — it is *where* you enforce the compliance boundary, not a compliance product you buy. An LLM gateway is a proxy that sits between your services and the model APIs, centralizing auth, routing, redaction, logging, and key management in one place instead of scattering them across every service that calls an LLM. For regulated AI the real question is not managed versus self-hosted in the abstract; it is whether the place you enforce that boundary sits inside your compliance perimeter or inside a vendor's.

This is a deep dive under the [AI governance framework](/library/ai-governance-for-regulated-products) hub. That framework names five controls that keep AI features inside your SOC 2 or HIPAA boundary; the gateway is the piece of infrastructure where four of them actually get enforced. If you are still deciding whether a given feature can ship at all, start with the pre-flight in [adding AI features without breaking your compliance posture](/library/ai-features-compliance-posture) and come back here to decide where the controls live.

## What is an LLM gateway?

An LLM gateway is a reverse proxy in front of your model providers that centralizes the cross-cutting concerns every AI feature needs: authentication and per-app internal keys, routing and failover across providers, rate limiting, cost and token tracking, request/response logging, PII/PHI redaction, and a single provider-agnostic API. Instead of each service holding its own OpenAI or Anthropic key and implementing its own logging, every service calls the gateway, and the gateway calls the provider.

It is the API-gateway pattern applied to model calls. The value is consolidation: one place that knows how to talk to every provider, one place where a prompt can be inspected before it egresses, one place where a model call is recorded, and one place where a leaked key can be rotated without a redeploy of six services. In a system with a single AI feature you may not feel the need. By the fifth feature — a summarizer, a classifier, a draft-generator, a support-bot, an embedding pipeline — the alternative to a gateway is the same four controls implemented five times, slightly differently, with one of them wrong.

Popular self-hostable options include LiteLLM and Portkey's open-source proxy; managed and SaaS gateways, plus the model platforms' own gateway features, cover the same surface as a service. Feature sets and pricing move fast in this category, so treat any specific capability as list-and-approximate and verify the current split before you commit.

## What does an LLM gateway do for compliance?

For regulated AI, the gateway is the single chokepoint where you enforce the compliance boundary your governance framework defines — which is exactly why *where it runs* matters more than which product it is. Four jobs sit here, and each maps to one of the leak points an LLM feature opens:

- **Scrub PII/PHI before it leaves your boundary.** The gateway is the last hop you control before the prompt reaches a third party, so it is the natural place to apply minimum-necessary redaction — strip or tokenize the fields the model does not need to see. This only works if the scrub runs before egress, which is to say inside your perimeter.
- **Restrict egress to BAA-covered providers only.** A gateway can refuse any destination you have not covered with a Business Associate Agreement (or the DPA your privacy commitments require). That turns "we think everyone uses the approved provider" into an enforced allowlist — the difference between a policy and a control.
- **Log inputs and outputs for audit — content-free.** The gateway emits one record per call so you can answer "what model saw what, when, and who approved the output" without keeping a second copy of the data. Log the shape, not the payload; the trace schema and the reason full prompt logs are the *wrong* answer are in [proving your AI features work: auditability for LLM systems](/library/llm-feature-auditability).
- **Rotate provider keys centrally.** One custody point for OpenAI, Anthropic, and every other provider key means rotation is a single operation, not a scavenger hunt across services — the same discipline as [killing long-lived credentials](/library/killing-long-lived-credentials), applied to model providers. Downstream services hold an internal gateway key, not the provider's, so a leaked internal key never reaches the provider directly.

The catch is that every one of those controls is only as good as its location. Redaction that runs after the data has already reached a third party is not redaction. A log store you do not control is an out-of-boundary regulated-data store. That is the whole self-hosted-versus-managed question in one sentence.

## Self-hosted vs. managed vs. no gateway: how do they compare?

Self-hosted keeps data, logs, and keys inside your VPC at the cost of running the proxy; managed offloads the ops but makes the gateway vendor a new sub-processor that sees your prompts; no gateway avoids both but duplicates every control across every service and gives you no chokepoint. The table is the decision in one view.

| Dimension | Self-hosted gateway (in your VPC) | Managed / SaaS gateway | No gateway (direct SDK calls) |
|---|---|---|---|
| Data residency | Prompts, responses, and logs stay inside your VPC and compliance boundary; nothing new leaves | Prompts transit the vendor's infrastructure; residency is wherever they run it | Data goes straight to each provider; boundary logic is scattered across every service that calls out |
| Added sub-processors | None beyond the model providers you already vetted | The gateway vendor becomes a new sub-processor that sees your prompts — disclose, DPA/BAA, score it | Only the model providers, but any team can add a new one silently |
| Ops burden | You own it: deploy, patch, scale, monitor, and page on the proxy | Low; the vendor runs it | None to run, but the compliance work is re-implemented in every service |
| Auth / key management | Provider keys held in one place in your boundary; rotate centrally; issue internal per-app keys | Provider keys live with the vendor, or you hand them over — a key-custody question to answer explicitly | Keys sprinkled across services and secret stores; the long-lived-credential problem multiplied |
| PII / PHI redaction | Runs inside your boundary *before* egress — the scrub happens where you control it | Runs on the vendor's side, so raw PII/PHI reaches them before it is scrubbed (defeats the purpose for PHI) | Each service must implement its own redaction; easy to miss one, and no one place to verify |
| Logging / audit | Content-free traces to your own evidence store; you own retention and access | Logs live in the vendor's dashboard by default, often as full prompts/responses — an out-of-boundary data store | Inconsistent per service; no single audit chokepoint to query |
| Provider abstraction / failover | One API for many providers; routing and failover you control | One API for many providers; routing managed for you | Each provider's SDK wired directly; switching or failover is a code change |

Read the redaction and logging rows together, because they are the crux. A managed gateway that runs redaction on its side has already received the raw PHI by the time it scrubs — which is fine for de-identified traffic and disqualifying for protected data unless that vendor is itself under a BAA. The residency and sub-processor rows are the same point from the data-flow angle: a managed gateway moves your PHI/PII boundary outward to wherever the vendor runs, exactly the outbound edge the [governance framework](/library/ai-governance-for-regulated-products) tells you to map before you ship.

## Is a managed AI gateway a sub-processor?

Yes — a managed or SaaS gateway sees every prompt and response you route through it, which makes it a sub-processor processing your data on your behalf, exactly like the model provider behind it. It is the easy one to forget, because it feels like plumbing rather than a data processor. It is both.

Treat it like any vendor with standing access to sensitive data: disclose it on your public sub-processor list, sign the DPA, and run it through [the vendor risk scoring rubric](/library/vendor-risk-scoring) before access is granted — a gateway that can read prompts built from patient records scores high on data sensitivity and access depth, which is precisely the vendor that earns a real review. If PHI flows through it, you need a BAA with the gateway vendor too, not only with the model provider, because the gateway is handling PHI in its own right. An undisclosed AI gateway discovered mid-review reads the same way an undisclosed model vendor does: like you lost track of where the data goes.

There is a middle path that dodges the new-sub-processor problem: **Amazon Bedrock**. It is managed — AWS runs the models and the routing, so you skip the ops of a self-hosted proxy — but it sits inside AWS, under your existing AWS BAA, which covers its HIPAA-eligible services (verify current eligibility, as the list shifts). Because it is already in your AWS boundary, using Bedrock does not add a new external sub-processor the way a third-party SaaS gateway does. It is the managed-but-in-boundary option, and for many regulated SMBs already on AWS it is the shortest path to a compliant setup. You still put your own thin gateway or middleware in front of it for redaction, internal keys, and content-free logging — Bedrock is the model boundary, not the whole control set.

## Which should you pick?

Pick a self-hosted in-VPC gateway (or Bedrock under a BAA) when regulated data hits the prompt, and a managed gateway when it does not. That is the verdict, and it follows directly from the residency and redaction rows above: for PHI or other regulated data, the redaction, the logs, and the provider keys have to stay inside your perimeter, which a self-hosted proxy or Bedrock gives you and an external SaaS gateway does not. For de-identified, synthetic, or genuinely low-stakes traffic — internal tooling, public-data summarization — a managed gateway is a fine trade: you get the convenience and you pay for it with one more disclosed, scored sub-processor.

Going without a gateway at all is defensible only at one or two AI features. Past that, the absence of a chokepoint *is* the risk: no single place to enforce the BAA allowlist, no single place to verify redaction ran, no single trace store to hand an auditor. The centralization is the point, and the point arrives fast.

The good-enough bar: a regulated SMB has this right when it can name one place where PII/PHI is scrubbed before egress, where provider keys are rotated, and where every model call is logged content-free — and can say honestly whether that place is inside its boundary or a vendor's. If the answer is "inside ours" (a self-hosted proxy or Bedrock), you clear the bar for regulated data. If it is "a managed vendor's," you clear it for de-identified data provided the vendor is disclosed and scored. The failing answer is the one most teams actually have: redaction and key handling copy-pasted across five services, no chokepoint, and no one who can say where the March prompts went.

## The Template

Two parts: decide whether and where the gateway lives, then confirm the controls actually run there. Score each item honestly. Any `no` in the control section blocks regulated traffic through that path until it is resolved.

**Do you need one? (gateway trigger)**
- [ ] More than one or two features send data to a model provider
- [ ] Any feature sends PHI, PII, or other regulated data into a prompt
- [ ] Provider keys currently live in more than one service or secret store
- [ ] You cannot today answer "what model saw what, when" from one place
- [ ] Two or more of the above true → stand up a gateway; regulated data true → it must be in-boundary

**Where does it live? (build-vs-buy)**
- [ ] Regulated data in prompts → self-host in-VPC, or use Bedrock under your BAA
- [ ] De-identified / low-stakes only → managed gateway acceptable if disclosed + scored
- [ ] Already on AWS and want managed → evaluate Bedrock before a third-party SaaS gateway
- [ ] If self-hosting: a named owner can deploy, patch, scale, and get paged on the proxy
- [ ] If managed: vendor scored on the vendor-risk rubric, on the sub-processor list, DPA signed (BAA if PHI)

**Controls that must run at the gateway**
- [ ] PII/PHI redaction runs *before* egress, inside your boundary (not on a vendor's side for PHI)
- [ ] Egress allowlist enforces BAA/DPA-covered providers only — unapproved destinations refused
- [ ] Provider keys held once, in-boundary; rotated centrally; downstream services hold internal keys only
- [ ] Every call emits a content-free trace (ids, versions, hashes, flags, decision) to your own evidence store
- [ ] Prompts/responses are NOT logged as content to the gateway's or a vendor's default dashboard
- [ ] Model versions pinned (dated snapshots, not moving aliases) and recorded per request
- [ ] No silent fallback to a hosted tier that would route prompts back outside the perimeter

**Prove it**
- [ ] The one place data is scrubbed, keys rotate, and calls are logged is named and documented
- [ ] You can state, per feature, whether that place is inside your boundary or a vendor's
- [ ] The gateway (self-hosted or managed) is in scope for the next access review + evidence pipeline
- [ ] A re-review triggers when a provider is added, a vendor changes terms, or the data flow changes

Most SMBs find the trigger section mostly true by their third or fourth AI feature and land on a self-hosted proxy in-VPC or Bedrock for anything touching regulated data, with a managed gateway reserved for the de-identified lane. All three shapes are defensible for the right data class. The one indefensible position is having no chokepoint at all once the AI features multiply — because that is the position where you cannot answer where the data went.

## Frequently asked questions

**Q: Do you need an LLM gateway for compliance?**

You need one the moment more than a feature or two sends regulated data to a model provider, but the gateway does not make you compliant on its own. It is where you enforce the compliance boundary your AI governance framework already defines: it is the single chokepoint where PII/PHI gets scrubbed before it leaves your systems, where traffic is restricted to BAA-covered providers, where every call is logged content-free for audit, and where provider keys are rotated centrally. Without a gateway those four controls get copy-pasted into every service that calls an LLM, and the one service that skips redaction is the one that ends up in a breach notification. The gateway is not a compliance purchase; it is the place your existing access, logging, and change-management controls get applied to model calls.

**Q: What is an LLM gateway?**

An LLM gateway is a reverse proxy that sits between your services and the model provider APIs and centralizes the cross-cutting concerns every AI feature needs: authentication and per-app internal keys, routing and failover across providers, rate limiting, cost and token tracking, request/response logging, PII/PHI redaction, and a single provider-agnostic API. It is the API-gateway pattern applied to model calls. Instead of each service holding its own OpenAI or Anthropic key and implementing its own logging, every service calls the gateway and the gateway calls the provider, so the controls live in one place you can audit. Self-hostable options include LiteLLM and Portkey's open-source proxy; managed and SaaS gateways cover the same surface as a service.

**Q: Self-hosted vs managed LLM gateway — which for regulated AI?**

For prompts carrying PHI or other regulated data, self-host the gateway inside your VPC (or use Amazon Bedrock under a BAA) so prompts, logs, and provider keys never leave your compliance boundary; you pay for that with the ops burden of running the proxy. A managed or SaaS gateway is the right call for de-identified or low-stakes traffic where the convenience is worth adding one more sub-processor to your list. The deciding question is never which product has more features — it is whether the place your data gets redacted and logged sits inside your perimeter or inside a vendor's. Redaction that runs after the raw data has already reached a third party is not redaction.

**Q: How do you make LLM features HIPAA compliant?**

Sign a BAA with the model provider (or de-identify before the data leaves your systems), keep PHI on HIPAA-eligible services only, log the interaction content-free, and put a human in the loop on consequential output — and route it all through a gateway that lives inside your HIPAA boundary so those controls are enforced in one place. The gateway is what makes the HIPAA controls structural rather than per-service good intentions: it scrubs the minimum-necessary fields before egress, refuses any provider you have not covered with a BAA, and emits a content-free trace to your evidence store. A managed gateway that sees raw PHI needs its own BAA, not just the model provider's, because it is processing PHI on your behalf too.

**Q: Is a managed AI gateway a sub-processor?**

Yes. A managed or SaaS gateway sees every prompt and response you route through it, which makes it a sub-processor processing your data on your behalf, exactly like the model provider behind it — and one that is easy to forget because it feels like plumbing. Disclose it on your public sub-processor list, sign the DPA (and a BAA if PHI flows through it), and score it on your vendor-risk rubric before access is granted. An undisclosed AI gateway discovered during a customer security review reads the same way an undisclosed model vendor does: like you lost track of where the data goes. Amazon Bedrock is the exception that proves the rule — it is managed but runs inside AWS under your existing BAA, so it is not a new external sub-processor.

**Q: LiteLLM vs Portkey — which self-hosted gateway?**

Both ship open-source, self-hostable proxies that cover the core gateway surface — a unified API across providers, key management, logging, and routing — so the choice comes down to fit rather than capability, and you should verify the current feature split before committing because both move fast. LiteLLM is widely used as a lightweight proxy and Python SDK that normalizes many providers behind one OpenAI-style API; Portkey's open-source gateway emphasizes routing, guardrails, and observability, with a managed tier alongside it. Treat any specific feature or price here as list-and-approximate. For regulated AI the load-bearing decision is not which of the two you pick — it is that you run it inside your boundary, wire its logs to your own content-free trace store, and do not silently fall back to their hosted tier, which would put your prompts back outside the perimeter.

Template: "LLM gateway decision + control checklist" — branded PDF at https://kaansystems.com/templates/llm-gateway-regulated-ai.pdf

Source: https://kaansystems.com/library/llm-gateway-regulated-ai · Work with Kaan Systems: https://kaansystems.com/contact

---

# Platform Engineering for Funded SaaS, Health-Tech, and Fintech

Canonical: https://kaansystems.com/library/platform-engineering-for-regulated-startups
Topic: Platform · Published: 2026-09-16 · Updated: 2026-10-29
Summary: Platform engineering for funded SaaS, health-tech, and fintech is four jobs: the AWS foundation, the delivery path, cluster reliability, and ongoing control. This is the Terraform hub. Audit evidence falls out of running the jobs, not out of a separate program.

## Key takeaways

- Platform engineering for a startup is four jobs, not a headcount: the AWS account foundation, the delivery path, cluster reliability, and ongoing control (drift + identity). Sequence them; each makes the next meaningful.
- You almost never need a dedicated platform team at 10-80 engineers — you need the platform. One senior owner (staff or fractional) plus paved-road defaults beats a team standing up bespoke tooling.
- Foundation first: a real AWS account boundary that isolates production is the control everything else assumes. Tag-based separation in one account is an audit finding, not a boundary.
- For most funded SaaS, health-tech, and fintech teams, buy the managed baseline (Control Tower, a managed GitOps controller) and spend your scarce senior time on the controls a vendor can't ship: identity, least privilege, and evidence.
- The platform is where most of your SOC 2 and HIPAA controls actually live — access, change management, logging, isolation. Built right, the evidence falls out of running the system instead of a quarterly screenshot drill.
- The good-enough bar, not the enterprise bar: a leaked credential in your least-trusted environment can't reach production data, git is the only write path to prod, and dangerous drift pages while the rest goes to a digest.

**What is platform engineering for funded SaaS, health-tech, and fintech, and what do you actually have to build?** Platform engineering is the discipline of building the internal foundation your product teams ship on — the cloud accounts, the delivery path, the clusters, the guardrails — so that shipping *safely* is the path of least resistance. For a funded SaaS, health-tech, or fintech team of 10-80 engineers it comes down to four jobs: a sound **foundation** (your AWS account structure), a disciplined **delivery** path (how code reaches production), real **reliability** (clusters that clear a production bar), and ongoing **control** (drift and credentials). Get those four right, in order, and you've also built most of your SOC 2 or HIPAA controls — because the platform is where those controls physically live. This guide is the map; the linked deep dives are the terrain.

A note on scope before we start. This is written for funded SaaS, health-tech, and fintech: SOC 2- or HIPAA-bound, ten to eighty engineers, a real product with real customer data. Not for the two-person prototype (you're not here yet) and not for the 500-engineer org with a dozen product teams (your platform is a staffed product with its own roadmap). In the middle band, the enemy isn't under-building. It's buying the enterprise version of everything before you've done the sequencing that makes it worth anything.

## Do you need a dedicated platform team to do this?

Almost never at this size — you need the *platform*, which is a different purchase. A dedicated platform team earns its coordination cost once you have enough product teams that a shared, self-serve internal platform pays for itself. Below that, the winning shape is one senior owner — a staff engineer who takes the mandate, or a fractional platform engineer — who builds paved-road defaults everyone uses and resists the urge to build a bespoke internal developer platform nobody asked for.

The failure mode isn't under-investment; it's fragmentation. Every product team inventing its own AWS layout, its own deploy script, its own way of handling secrets is how a 40-person company acquires the governance problem of a 400-person one, minus the staff to manage it. One person owning the four jobs below, with opinionated defaults, beats a committee every time.

## What are the four jobs, and what does "good enough" look like for each?

The whole map is on one page. Each job has an honest "good enough" bar for a funded SaaS, health-tech, or fintech team, deliberately not the enterprise bar, and a deep dive that builds it. Work them top to bottom; a later job built on a missing earlier one is wasted money.

| The platform's job | What "good enough" looks like | The deep dive |
|---|---|---|
| **1. Foundation** — AWS account structure | A real account boundary isolates production; logs are tamper-resistant; one identity plane fronts access. A leaked staging credential can't reach prod data. | [Building a multi-account landing zone](/library/multi-account-aws-landing-zone) |
| **2. Delivery** — how code ships | Git is the only write path to prod; deploys are reviewed and gated; rollback is automatic; the git log *is* your change-management evidence. | [GitOps + progressive delivery](/library/gitops-progressive-delivery) |
| **3. Reliability** — does it stay up | The cluster clears a concrete production-readiness bar, not a vibe. Incidents are debuggable at 3 AM by someone who didn't build it. | [What "production-ready" actually means](/library/production-ready-kubernetes) |
| **4. Control** — does it stay correct + least-privilege | Dangerous drift pages; the rest is a weekly digest. No long-lived credentials; access is short-lived and federated. | [Drift detection](/library/infrastructure-drift-detection) + [killing long-lived credentials](/library/killing-long-lived-credentials) |

The rest of this guide is one section per job.

## Job 1 — Foundation: what AWS account structure does a funded SaaS need?

Start here, because every other control assumes it. The foundation is your AWS account topology, and for a regulated team the load-bearing property is *isolation you can point at*: production separated from everything else by a boundary an auditor accepts, logs an operator can't tamper with, and access that flows through a single identity plane.

Three decisions live in this job, and they're sequential.

**First: do you even need a landing zone yet?** Not on day one. A single AWS account is the right, boring choice for an early prototype, and building the full topology before you have anything to protect is premature optimization you pay interest on. The honest triggers are specific — real customer data or PHI in production, a SOC 2 or HIPAA requirement, a second environment, or more than a couple of engineers — and you read them as an OR. [Do you actually need an AWS landing zone?](/library/when-do-you-need-aws-landing-zone) walks the decision tree and the tipping points; if you are funded SaaS, health-tech, or fintech reading this, you have almost certainly crossed at least one.

**Second: how do you build it without over-building?** The minimum viable landing zone is four to five accounts — a management account that runs nothing, a log-archive account with Object Lock so evidence is undeletable, a security/audit account, and separate prod and non-prod workload accounts — not the twenty-account enterprise build. [Building a multi-account AWS landing zone](/library/multi-account-aws-landing-zone) covers the OU topology, a starter service-control-policy set, identity, and the retrofit sequence that lets you invite an existing production account into the org *without* a migration weekend.

**Third: buy or build the tooling?** This is a genuine build-vs-buy call, and for most funded SaaS, health-tech, and fintech teams the answer is buy. AWS Control Tower hands you managed guardrails, centralized logging, and audit-ready raw material in an afternoon; raw Terraform gives you total control and multi-cloud portability at the cost of owning all of it. [Control Tower vs. Terraform](/library/control-tower-vs-terraform-landing-zone) makes the call dimension by dimension — the short version is Control Tower (or Control Tower plus Account Factory for Terraform) unless you're multi-cloud or already run a Terraform monorepo with a platform engineer to own it.

**Good-enough bar:** a leaked credential in your least-trusted environment cannot reach production data, and you can answer "how is prod isolated?" with a structural boundary rather than a tag you hope holds.

## Job 2 — Delivery: how should a funded SaaS ship code to production?

Through git, and only through git. The delivery job is about making the deploy path both fast *and* auditable — which are not in tension, despite how they get framed. The pattern that resolves it is GitOps: a controller pulls desired state from a git repository and reconciles the cluster to match, so nobody `kubectl apply`s into production by hand and no human holds standing cluster-admin credentials.

The regulated payoff is the audit trail. When git is the only write path, the git history becomes your change-management evidence — every production change is a reviewed, approved, timestamped commit, which is exactly what a SOC 2 or HIPAA assessor wants to see, produced as a byproduct of how you already work instead of a screenshot someone takes the week before the audit. Add SLO-gated progressive delivery — canary rollouts that promote or auto-roll-back on real service metrics — and human-gated deploys stop being a ceremony that slows you down and become a control that speeds you up. [GitOps and progressive delivery with automated rollback](/library/gitops-progressive-delivery) covers the repo structure, sync policies, and the canary setup that writes its own audit trail.

The one decision inside this job is which controller. It's a smaller decision than the six-week debate it usually triggers, because both mainstream options produce the same git audit trail and both auto-heal drift. [Argo CD vs. Flux](/library/argo-cd-vs-flux) settles it: pick Argo CD for a first-class UI and fine-grained multi-team RBAC (the default for most of these teams, because walking an auditor through a live console beats assembling evidence from logs), and pick Flux for a lean, composable, CLI-first footprint a small platform team owns. The wrong move is leaving the choice open for a quarter.

**Good-enough bar:** git is the only write path to production, deploys are reviewed and SLO-gated, rollback is automatic, and you can produce the change history for any production change without touching a screenshot tool.

[Book a call →](https://cal.com/benrubert/30min?utm_source=library&utm_medium=article&utm_campaign=platform-engineering-for-regulated-startups&utm_content=middle)

## Job 3 — Reliability: what makes a Kubernetes cluster production-ready?

"Production-ready" is a checklist, not a property — and the ambiguity around the phrase is where reliability quietly fails. Engineering says the cluster is production-ready meaning "it runs the app"; the auditor and the on-call engineer mean something much more specific, and the gap between those definitions is what turns into a 3 AM incident nobody can debug.

The fix is to make the definition concrete and score it honestly. [What "production-ready" actually means for a Kubernetes cluster](/library/production-ready-kubernetes) is a forty-item scorecard across five dimensions — scored 0-2, no partial credit for good intentions — covering the things that separate "it runs" from "it survives contact with reality": resource limits and autoscaling, network policy and identity, observability that makes incidents debuggable, backup and tested restore, and the security posture an auditor probes. Most teams score around 50 on the first honest pass, and the next fifteen points is where the real architectural work lives.

For a regulated team, several scorecard items double as compliance controls — network policy is isolation, workload identity is least privilege, centralized logging is your audit trail — which is the recurring theme of this guide: the reliability work and the compliance work are largely the same work.

**Good-enough bar:** the cluster clears a concrete, written production-readiness bar (not a vibe), and an engineer who didn't build it can debug an incident at 3 AM from the observability alone.

## Job 4 — Control: how do you keep the platform correct and least-privilege over time?

The first three jobs build the platform; the fourth keeps it honest as it drifts and as people come and go. Two controls matter most for a regulated team, and both are about the gap between "how it was set up" and "how it actually is right now."

**Drift detection.** Infrastructure declared in code drifts — someone clicks in the console during an incident, a resource gets modified out-of-band, and now your Terraform and your reality disagree. The naive version of drift detection drowns you: thousands of "differences," almost all of them noise, until everyone mutes the alerts and the one that mattered slips through. The discipline is tiering drift by risk so the dangerous 5% (a security group opened to the world, a disabled log) pages someone while the harmless 95% (a tag mismatch) goes to a weekly digest. [Detecting infrastructure drift without drowning in alerts](/library/infrastructure-drift-detection) is the classifier and the workflow — and for a regulated team, unreviewed drift is exactly the finding an auditor is looking for.

**Identity and short-lived access.** The highest-leverage control in the entire platform is eliminating long-lived credentials. Static keys — IAM user access keys, CI secrets, database passwords sitting in a config — are the breach vector that actually gets used, because they work until someone revokes them and nobody ever revokes them. The move is to replace stored secrets with workload identity: OIDC federation for CI, IRSA for pods in the cluster, and dynamic short-lived secrets for databases, with the count of remaining static credentials as the program's KPI. [Killing long-lived credentials with workload identity and dynamic secrets](/library/killing-long-lived-credentials) is the phased migration plan that does it on a live estate without breaking Friday's deploy. Auditors and attackers care about the same thing here, which is why this one control pays down both risk and evidence burden at once.

**Good-enough bar:** dangerous drift pages and everything else is a reviewed digest; and your IAM credential report has no long-lived keys nobody can explain, because access is short-lived and federated by default.

## How does all of this earn your SOC 2 or HIPAA?

Because most of the controls those audits test are platform controls — so if you've built the four jobs above, you've built the compliance program, not a parallel one. Environment isolation is Job 1. Change management is Job 2's git history. Reliability and centralized logging are Job 3. Least privilege and access control are Job 4. The platform either enforces these or it doesn't; a binder of policies describing controls that don't operate fails a Type II audit and, worse, doesn't make you secure.

The strategic point is *evidence as a byproduct*. Built right, the account boundary is the isolation control and its own evidence; the git log is the change-management control and its own evidence; the identity plane is the access control and its own evidence. That's the difference between a first audit that consumes a quarter of screenshots and a renewal that's boring. If SOC 2 specifically is what's driving this, our [SOC 2 for startups complete guide](/library/soc2-for-startups-complete-guide) maps the controls to the report — and you'll notice most of its remediation phase *is* the four jobs on this page.

## The verdict: sequence over spend

The mistake that sinks platform work at a funded SaaS, health-tech, or fintech company isn't buying too little — it's buying the enterprise version of job three while job one is still open. A microsegmentation product on a flat account structure, a service mesh on a cluster that fails its own readiness scorecard, a compliance platform pointed at controls that don't exist yet. Sequence is the whole discipline: foundation, then delivery, then reliability, then control. Each job makes the next one meaningful, and the good-enough bar at each is reachable this quarter with tooling you mostly buy rather than build. Spend your scarce senior time on the parts a vendor can't ship for you — identity, least privilege, and evidence — and let the managed baselines handle the rest.

## Where is the Terraform hub?

This page is it. Foundation, delivery, reliability, and control are the four jobs. The links above are the build. The [Terraform hub](/library/terraform) lists them in one place: the landing-zone threshold, the multi-account design, Control Tower versus Terraform, drift, GitOps, the controller choice, the production-readiness scorecard, credentials, and evidence.

If production is still changed in the console, start with [from click-ops to code](/library/click-ops-to-terraform). The [7-day Assessment](/assessment) is the read-only edition of that review: seven business days, including a drift scan.

## The Template

A platform foundation readiness checklist. Work top to bottom, one job at a time — a later job built on an incomplete earlier one is wasted money. Score each item Not started / In progress / Done.

**Job 0 — Ownership**
- [ ] One named senior owner for the platform (staff or fractional), not a committee
- [ ] Paved-road defaults are the documented path, not one team's private setup
- [ ] Platform work has a place on the roadmap, not just the gaps between features

**Job 1 — Foundation (AWS account structure)**
- [ ] Confirmed you've crossed a real landing-zone trigger (data / compliance / second env / team size)
- [ ] Multi-account landing zone stood up (min viable: management, log-archive, security, prod, non-prod)
- [ ] Log-archive account with Object Lock — logs are undeletable
- [ ] Build-vs-buy call made (Control Tower vs. Terraform) and recorded, not left open
- [ ] Production isolated by an account boundary, not a tag: a staging credential can't reach prod data

**Job 2 — Delivery (how code ships)**
- [ ] GitOps controller reconciling prod from git — no hand-run `kubectl apply`
- [ ] Git is the only write path to production; no standing human cluster-admin
- [ ] Controller choice (Argo CD vs. Flux) made and wired to SSO + RBAC
- [ ] Progressive delivery: SLO-gated canaries with automatic rollback
- [ ] Change history for any prod change is produceable without a screenshot tool

**Job 3 — Reliability (does it stay up)**
- [ ] Cluster scored against a written production-readiness checklist (not a vibe)
- [ ] Resource limits, autoscaling, and network policy in place
- [ ] Observability makes an incident debuggable by someone who didn't build it
- [ ] Backup + a *tested* restore, not just a backup job that's green
- [ ] First honest score recorded; the gap to the target is on the roadmap

**Job 4 — Control (correct + least-privilege over time)**
- [ ] Drift detection tiered by risk: dangerous drift pages, the rest is a digest
- [ ] Long-lived credential inventory taken; the count is a tracked KPI
- [ ] CI uses OIDC federation; pods use workload identity (IRSA); DBs use dynamic secrets
- [ ] No unexplained static keys in the IAM credential report
- [ ] Access is short-lived and federated by default

**The compliance byproduct**
- [ ] Account boundary doubles as the isolation control + its evidence
- [ ] Git history doubles as the change-management control + its evidence
- [ ] Identity plane doubles as the access control + its evidence
- [ ] Evidence collection is a byproduct of running the system, not a quarterly drill

Most teams find Job 1 is the unlock and Job 4 is where discipline pays off or quietly rots. Both are engineering problems more than compliance ones — which is the whole point.

## Frequently asked questions

**Q: What is platform engineering for startups?**

Platform engineering is the discipline of building the internal foundation your product teams ship on — the cloud accounts, the deploy pipeline, the clusters, and the guardrails — so that shipping safely is the path of least resistance. For a startup, especially a regulated one, it's less about a fancy internal developer platform and more about four concrete jobs: a sound AWS account foundation, a delivery path where git is the only write path to production, clusters that clear a real production-readiness bar, and ongoing control over drift and credentials. Done well, it's also where most of your SOC 2 or HIPAA controls physically live, so the platform work and the compliance work are the same work.

**Q: Do you need a dedicated platform team?**

Usually not at 10-80 engineers — you need the platform, which is a different thing. A dedicated team is what you build once product teams are large enough that a shared, self-serve platform pays for the coordination cost, typically past the point covered by this guide. Before then, the winning shape is one senior owner (a staff engineer or a fractional platform engineer) who builds paved-road defaults everyone uses: a landing zone, a GitOps deploy repo, a production-readiness scorecard. What you want to avoid is the opposite failure — every product team inventing its own AWS layout, deploy script, and secret handling, which is how a 40-person company ends up with the governance problem of a 400-person one.

**Q: When do you need an AWS landing zone?**

Not on day one — a single AWS account is the correct, boring choice for an early prototype. You need a landing zone the moment you cross the first tipping point: real customer data (or PHI/PII) in production, a SOC 2 or HIPAA requirement, a second environment, or more than two or three engineers. Read that as an OR, not an AND; most teams hit two of them within a quarter. The trigger is a real deal or obligation, not a feeling that you should be more grown-up, and the minimum viable version is four or five accounts, not the twenty-account enterprise build.

**Q: Is GitOps worth it for a small team?**

Yes, and arguably more so for a small regulated team than a large one, because it replaces three problems at once with one pattern. A pull-based GitOps controller reconciling the cluster to state declared in git kills configuration drift, makes rollback a git revert instead of an archaeology project, and — the part that matters for compliance — produces a change-management audit trail as a byproduct of how you already deploy. For a team without a dedicated release engineer, the git history becomes evidence an auditor accepts without a single manual screenshot. The controller choice (Argo CD vs. Flux) is secondary to adopting the pattern at all.

**Q: How does platform engineering help with SOC 2 or HIPAA?**

Most of the controls a SOC 2 or HIPAA audit tests are platform controls, so the platform work and the compliance work overlap heavily. Environment isolation, least-privilege access, change management, centralized tamper-resistant logging, and encryption are all things the platform either enforces or it doesn't. Built correctly, the platform doesn't just satisfy the controls — it generates the evidence continuously: the account boundary is the isolation control, the git history is the change-management control, the identity plane is the access control. That turns audit evidence from a quarterly fire drill into a byproduct of running the system, which is the difference between a painful first audit and a boring renewal.

**Q: Platform engineering vs. DevOps — what's the difference?**

DevOps is a culture — dev and ops sharing ownership of shipping and running software; platform engineering is the concrete product that culture produces at scale: a paved road that makes the right way the easy way. In practice, 'DevOps' at a startup often means a couple of engineers hand-tuning pipelines and clicking around the AWS console, while platform engineering means codifying that into a landing zone, a GitOps repo, and guardrails other engineers self-serve against. They're not in conflict; platform engineering is what mature DevOps looks like once you stop doing it by hand. For funded SaaS, health-tech, and fintech, the platform-engineering framing matters because guardrails and evidence are first-class deliverables, not afterthoughts.

Template: "Platform foundation readiness checklist" — branded PDF at https://kaansystems.com/templates/platform-engineering-for-regulated-startups.pdf

Source: https://kaansystems.com/library/platform-engineering-for-regulated-startups · Work with Kaan Systems: https://kaansystems.com/contact

---

# Rightsizing AWS Without Breaking Production: A Data-Driven Playbook

Canonical: https://kaansystems.com/library/rightsizing-aws-without-breaking-production
Topic: FinOps · Published: 2026-09-16 · Updated: 2026-10-12
Summary: Rightsizing is matching resource size to actual utilization without hurting latency or errors — the safest cloud cost lever when you do it data-first, and an incident when you resize from a spreadsheet. Measure over a real business cycle, set a headroom bar, change one workload at a time behind a rollback, and take the free storage wins. Do it after measuring and before you commit to Savings Plans, because you never want to lock in waste.

## Key takeaways

- Rightsizing is matching resource size to real utilization without hurting latency or errors; the data-driven part is measuring first, changing one workload at a time, and keeping a tested rollback path.
- Measure over a representative window with CloudWatch plus Compute Optimizer, but memory needs the CloudWatch agent installed and T-family burstables need CPU-credit awareness, not just average CPU.
- Set a headroom policy before you read a single recommendation — target p95 CPU and memory under about 70% after the change — so you cut toward a bar you chose, not toward a tool's aggressive default.
- Change one workload at a time via a launch template or Auto Scaling group, and watch p95/p99 latency and error rate before and after — CPU alone will call a latency regression a success.
- Storage is the safest money: gp2 to gp3, delete unattached EBS, and lifecycle your snapshots and S3; do all rightsizing after measuring and before committing to Savings Plans, because you never want to commit to waste.

**How do you rightsize AWS instances without breaking production? Measure real utilization over a representative window, set a headroom target, then change one workload at a time behind a tested rollback while you watch latency and errors — not just CPU.** Rightsizing is matching resource size to how a workload actually behaves, and it goes wrong the moment someone resizes from a Compute Optimizer number instead of from the workload's traffic. Do it data-first and it is one of the safest cost levers you have; do it from a spreadsheet and it is how a cost review becomes an incident.

This is the deep version of the "right-size" week that [the 30-day cloud cost cleanup](/library/30-day-cloud-cost-cleanup) schedules but deliberately does not fully document — that guide gives you the calendar, this one gives you the safe method. It also belongs in a specific place in the sequence: rightsize *after* you have measured and *before* you commit to [Savings Plans or Reserved Instances](/library/savings-plans-vs-reserved-instances), because a one- or three-year commitment locks in whatever waste you failed to remove first.

## What is rightsizing, and why does it break production?

Rightsizing is matching a resource's size to its actual utilization without degrading performance — right CPU, right memory, right storage class for the load the thing really carries. It breaks production for one reason: people treat the recommendation as the answer. A tool reports 8% average CPU across a fleet, someone drops every instance two sizes, and the p95 that was hiding behind that 8% average now saturates. Latency climbs, the autoscaler thrashes, and the "cost win" turns into a postmortem.

The data-driven version replaces the average with the distribution, the whole fleet with one careful change, and the hope with a rollback. You are not cutting to a number. You are cutting to a headroom bar you decided in advance — and you can only do that honestly if you can [attribute each resource to a real workload](/library/cost-aware-tagging). Rightsizing the wrong service's instance just moves the outage around.

## How do you measure before you cut?

Measure over a window long enough to contain the workload's real peaks — a full business cycle, not a quiet weekend. Two sources feed the decision, and both have a trap.

CloudWatch is ground truth for CPU, network, and disk, but it does not see guest memory by default. Memory utilization requires the CloudWatch agent installed on the instance; without it you are rightsizing memory blind, and Compute Optimizer will fall back to defaults or report memory as unavailable. If memory is your binding constraint — and for most application and database workloads it is — install the agent first and let it collect before you trust any recommendation.

AWS Compute Optimizer reads those metrics and produces EC2, Auto Scaling group, EBS, Lambda, and RDS recommendations for free. Its default lookback is roughly 14 days; the longer three-month "enhanced infrastructure metrics" lookback is a paid opt-in (verify current pricing). Fourteen days is fine for a steady service and dangerous for anything with a monthly close, a payroll run, or a seasonal peak that lives outside the window.

The trap that catches everyone: T-family burstable instances (t3, t4g) are *supposed* to run at low average CPU. That is the design, not waste. A tool that sees 5% average CPU on a `t3.large` and suggests shrinking it can be flatly wrong — what matters is the CPU credit balance and whether the instance is quietly running in unlimited mode and billing for sustained bursts. Read the credits before you touch a burstable.

## Is AWS Compute Optimizer good enough to trust?

Compute Optimizer is a good starting point and a bad autopilot. It is free, native, pulls the right metrics, and its recommendations are a reasonable first draft — genuinely useful for surfacing over-provisioned instances you would never audit by hand. What it does not know is your workload's tolerance: it does not know that this service has a strict p99 latency SLO, that that one is a batch job where a slower run is fine, or that the 14-day window missed month-end. Treat its output as a ranked list of candidates to investigate, never as a change set to apply. The tool finds the suspects; you decide the verdict.

## How much headroom should you leave?

Set the headroom policy before you look at a single recommendation, so the tool proposes and your policy disposes. A defensible default for most regulated-SMB workloads: target steady-state p95 CPU and p95 memory under about 70% after the change, with the instance able to absorb its normal peak without leaning on autoscaling. Use p95, not the average — the average is what lies to you. Latency-sensitive services get more headroom (cut conservatively, one size at a time); batch, dev, and staging get less (cut aggressively, they can tolerate it). Writing the bar down turns rightsizing from an argument into an arithmetic check.

## How do you rightsize without downtime?

Change one workload at a time, behind infrastructure you can roll back, and watch the symptoms that actually matter. The safe method is the same shape every time.

| Step | Do this | The guardrail |
|---|---|---|
| 1. Measure | CloudWatch (plus the agent for memory) over a full business cycle; pull Compute Optimizer | No agent means no memory data — don't rightsize memory blind |
| 2. Set headroom | Target p95 CPU and memory under ~70% post-change; more for latency-SLO services | Decide the bar before you read any recommendation |
| 3. Change one thing | New size via launch template / ASG, one workload at a time | Never batch-resize a fleet from a single average |
| 4. Keep a rollback | Previous launch template version or snapshot ready to reapply | No rollback means don't touch stateful or prod |
| 5. Watch the right signals | p95/p99 latency and error rate, before and after, for a full cycle | CPU alone will call a latency regression a success |

Two rules keep this from becoming an incident: never rightsize a stateful or production system without a tested rollback and a watch window, and watch latency and error rate, not just CPU. A smaller instance that holds CPU under 70% but doubles p99 latency is a failed change, and the CPU dashboard will happily tell you it succeeded. Make the change in your launch template or Auto Scaling group and land it in your IaC — a resize done by hand in the console is exactly the "someone rightsized by hand" signal that [infrastructure drift detection](/library/infrastructure-drift-detection) routes to its weekly digest, and if you never backport it, the next `terraform apply` reverts your saving.

## What are the storage and database wins?

Storage is the safest money in the whole exercise, because most of it carries no performance risk at all. Rank it easy-to-hard:

- **gp2 → gp3 EBS.** gp3 is faster and roughly 20% cheaper at the same provisioned size, and it lets you set IOPS and throughput independently of capacity (verify current pricing for your region). The migration is an online volume modification in most cases. For most volumes it is a strictly better default with no downside, and there is no good reason to keep launching gp2.
- **Delete unattached EBS.** Volumes sitting in the `available` state are 100% waste. Snapshot anything recent, then delete the volume.
- **Snapshot and AMI hygiene.** Retire orphaned snapshots and unreferenced AMIs on a retention policy instead of hoarding them forever. "Keep the last 30 days plus month-end for a year" is a sane default.
- **S3 lifecycle and Intelligent-Tiering.** Move cold objects to cheaper classes on a lifecycle rule, or let Intelligent-Tiering shift them automatically for unpredictable access patterns (it adds a small per-object monitoring charge — worth it above a size threshold, not below).
- **RDS rightsizing.** The highest-risk item, so it goes last. Compute Optimizer covers RDS, and the gp2 → gp3 move applies here too, but an instance-class change is a maintenance-window event with a real failover. Treat production RDS like production compute: rollback and watch window, never a Friday-afternoon click.

## What's the good-enough bar?

The good-enough bar: every production instance sits under your headroom target at p95 with a tested rollback path, every EBS volume is gp3, nothing unattached is still billing, and every change landed in IaC rather than the console. You do not need per-workload perfection or a commercial rightsizing platform to clear that bar — CloudWatch, Compute Optimizer, and discipline get a 10-to-80-engineer shop there.

The sequencing is the part people skip. Rightsize first, then buy commitments. Buying a three-year Savings Plan over an un-rightsized fleet is committing to your own waste at a discount, which is worse than paying on-demand for waste you can still delete tomorrow. Measure, cut to the bar, then [commit with Savings Plans or Reserved Instances](/library/savings-plans-vs-reserved-instances) against the smaller, true baseline. Never commit to waste.

## The Template

Run through it in order. Each box has a clear "done" state; do not start the compute cuts before the measurement boxes are checked.

**Before you touch anything**
- [ ] CloudWatch agent installed wherever memory is the binding constraint
- [ ] Metrics collected over a full business cycle (month-end / peak included)
- [ ] Compute Optimizer enabled; recommendations pulled for EC2, ASG, EBS, RDS, Lambda
- [ ] Every candidate resource attributed to a named workload and owner
- [ ] Headroom policy written down (p95 CPU/mem under ~70%; more for latency-SLO services)
- [ ] T-family burstables checked for CPU-credit balance, not just average CPU

**Making the change (one workload at a time)**
- [ ] Change lives in the launch template / ASG / IaC, not a console click
- [ ] Previous version or snapshot confirmed as a working rollback
- [ ] Applied to one workload, in a low-traffic window for anything stateful
- [ ] p95/p99 latency and error rate watched before and after, for a full cycle
- [ ] Result backported to IaC so drift detection stays quiet

**Storage and database sweep**
- [ ] All gp2 volumes migrated to gp3
- [ ] Unattached EBS volumes snapshotted and deleted
- [ ] Orphaned snapshots and unreferenced AMIs retired on a retention policy
- [ ] S3 lifecycle or Intelligent-Tiering applied to cold data
- [ ] RDS rightsized last, treated as a production change (window + rollback)

**Only then**
- [ ] Steady-state baseline re-measured after the cuts
- [ ] Savings Plans / RIs sized against the new, smaller baseline — never the old one

## Operating Notes

Rightsizing is not a one-time event. Workloads grow back into their instances, someone launches a fresh gp2 volume, a new service ships over-provisioned "to be safe." Re-run the measurement pass on the same cadence as the [30-day cost cleanup](/library/30-day-cloud-cost-cleanup) — a couple of times a year — and the estate stays close to its true workload cost instead of drifting back up.

Keep the discipline even when the numbers are small. The habit that saves you is not the single dramatic cut; it is that every change goes through measurement, a rollback, and a watch window, so the day you rightsize something load-bearing, the muscle is already there. The teams that break production rightsizing are the ones who did it carefully ten times, got comfortable, and skipped the rollback on the eleventh.

## Frequently asked questions

**Q: How do you rightsize AWS instances?**

Rightsize AWS instances data-first: measure real utilization with CloudWatch (add the CloudWatch agent for memory) over a full business cycle, pull AWS Compute Optimizer recommendations, set a headroom target such as p95 CPU and memory under about 70 percent, then change one workload at a time using a launch template or Auto Scaling group with a tested rollback. Watch latency and error rate before and after, not just CPU, and land the change in your IaC so it does not revert. Take the low-risk storage wins (gp2 to gp3, delete unattached EBS) before you touch compute, and do all of it before committing to Savings Plans.

**Q: What is rightsizing?**

Rightsizing is matching a resource's size to its actual utilization without degrading performance — the right CPU, memory, and storage class for the load a workload really carries. It applies to EC2 instances, EBS volumes, RDS databases, and Lambda memory. Done from measured data it is one of the safest ways to cut a cloud bill; done from a raw recommendation it is a common way to cause an outage.

**Q: Is AWS Compute Optimizer good?**

Compute Optimizer is a good starting point and a poor autopilot. It is free, native, and pulls the right CloudWatch metrics, so its recommendations are a useful ranked list of over-provisioned candidates. But it does not know your latency SLOs, your batch-versus-realtime split, or whether its default roughly 14-day window missed month-end, and it cannot see guest memory unless the CloudWatch agent is installed. Treat its output as suspects to investigate, not a change set to apply.

**Q: gp2 vs gp3 — should I switch?**

For almost every volume, yes. gp3 is faster and roughly 20 percent cheaper than gp2 at the same size, and it lets you provision IOPS and throughput independently of capacity (verify current pricing for your region). The migration is an online volume modification in most cases, so there is rarely a reason to keep launching gp2. It is usually the single safest rightsizing move you can make.

**Q: How much can rightsizing save?**

It varies with how neglected the estate is, but rightsizing plus the adjacent storage cleanup is a large share of the 20 to 40 percent that a first structured cost cleanup typically recovers. The compute cuts, the gp2-to-gp3 switch, and the unattached-EBS deletes compound. The bigger lever is sequencing: rightsizing before you buy Savings Plans keeps you from committing one to three years to waste you could have deleted for free.

**Q: How do you rightsize without downtime?**

Change one workload at a time through a launch template or Auto Scaling group, never a batch resize of a whole fleet from a single average. Keep a tested rollback — the previous launch template version or a snapshot — and for stateful or production systems make the change in a low-traffic window. Watch p95/p99 latency and error rate for a full business cycle after the change, because CPU alone will call a latency regression a success. Never rightsize a stateful or production system without both a rollback and a watch window.

Template: "Data-driven rightsizing checklist" — branded PDF at https://kaansystems.com/templates/rightsizing-aws-without-breaking-production.pdf

Source: https://kaansystems.com/library/rightsizing-aws-without-breaking-production · Work with Kaan Systems: https://kaansystems.com/contact

---

# Savings Plans vs. Reserved Instances vs. On-Demand: The Commitment Decision

Canonical: https://kaansystems.com/library/savings-plans-vs-reserved-instances
Topic: FinOps · Published: 2026-09-16 · Updated: 2026-10-12
Summary: The commitment decision is a FinOps trap: sign for three years and you lock in whatever you're running, waste included. Savings Plans now beat Reserved Instances for compute for most regulated SMBs, but only after you rightsize and only across the stable part of the baseline. Here is the comparison, the coverage math, and the honest verdict.

## Key takeaways

- For compute (EC2, Fargate, Lambda), Savings Plans beat Reserved Instances for most regulated SMBs: you commit to a dollars-per-hour spend and keep flexibility across instance family, size, region, and OS. AWS steers most teams here.
- Rightsize before you commit. A commitment locks in whatever you are running, so a three-year term on an oversized fleet just buys three years of discounted waste.
- Cover roughly 70-80% of your stable baseline, not 100%. Leave spiky and uncertain usage on On-Demand (or Spot for interruptible batch) so you never owe an hourly commitment you outgrew — Savings Plans are not refundable or resellable.
- Compute SP (floats across family/size/region/OS, slightly lower discount) vs EC2 Instance SP (locked to a family+region, higher discount); Standard RI (locked, highest RI discount, sellable in the Marketplace) vs Convertible RI (exchangeable, lower discount, largely superseded by Savings Plans).
- The good-enough bar: 1-year terms when uncertain, 3-year only for genuinely durable workloads, partial-upfront as the default balance — and still buy Reserved Instances or reserved nodes for the non-compute services (RDS, ElastiCache, Redshift, OpenSearch) that Savings Plans do not cover.

**Savings Plans vs. Reserved Instances: which should you buy?** For compute — EC2, Fargate, and Lambda — Savings Plans win for most regulated SMBs, because you commit to a dollars-per-hour spend rather than to specific instances and keep flexibility across instance family, size, region, and operating system. Reserved Instances are the older mechanism, and AWS now steers most teams to Savings Plans; the place RIs still clearly belong is the services Savings Plans do not cover. On-Demand stays the right home for anything spiky or uncertain — it is the most expensive per hour and the most flexible, and you pay that premium on purpose.

This is the decision-stage companion to [the 30-day cloud cost cleanup](/library/30-day-cloud-cost-cleanup), whose fourth week is "commit." That plan tells you *when* to commit; this article settles *what to commit to*, and the trap it exists to prevent: signing a three-year deal before you have any business signing anything. The single most expensive mistake in this whole area is committing to waste, which is why the first move is not a purchase at all.

## What are On-Demand, Savings Plans, and Reserved Instances?

They are three ways to pay for the same compute, trading flexibility for discount. On-Demand is pay-as-you-go: the list rate, no commitment, turn it on and off whenever. Savings Plans and Reserved Instances are both commitments — you promise AWS a certain amount of usage for one or three years, and in exchange the per-hour rate drops. The difference between them, and the difference between the flavors within each, is *what exactly you are committing to*.

- **On-Demand** — the baseline. Highest price per hour, total flexibility, zero commitment. This is what everything costs before you optimize.
- **Compute Savings Plan** — you commit to a dollars-per-hour spend (for example, $10/hr) for 1 or 3 years. It floats: the discount applies to EC2 across any family, size, region, and OS, and to Fargate and Lambda too. Slightly lower discount than the locked options, in exchange for that flexibility.
- **EC2 Instance Savings Plan** — same dollars-per-hour commitment model, but locked to one instance family in one region (size and OS still float). Higher discount than the Compute plan, because you have given up most of the flexibility.
- **Standard Reserved Instance** — the older mechanism. You reserve a specific instance type for the term. Highest discount of the RI options, but locked; its one escape hatch is that you can sell it in the AWS Reserved Instance Marketplace if you no longer need it.
- **Convertible Reserved Instance** — an RI you can exchange for a different configuration during the term, at a lower discount, and which you cannot resell.

The mental model that matters: **Savings Plans commit to spend; Reserved Instances commit to instances.** Committing to spend is almost always the more forgiving promise for a team that is still rightsizing and still changing shape — which describes every SMB at 10 to 80 engineers.

## Savings Plans vs. Reserved Instances vs. On-Demand: how do they compare?

Discount rises as flexibility falls; the term length multiplies both the discount and the risk. The table is the whole decision in one view. Treat every discount figure as AWS's headline "up to" number — it assumes the most aggressive term and payment on specific families, and your blended, real-world discount will be lower. Verify current pricing before you model anything.

| Option | Discount vs On-Demand | Flexibility | Term | Best for |
|---|---|---|---|---|
| **On-Demand** | 0% (baseline, highest price) | Total — any instance, any time, no commitment | None | Spiky, short-lived, or uncertain workloads; anything you might turn off next month |
| **Compute Savings Plan** | up to ~66% (approximate, verify current) | High — floats across family, size, region, OS; covers EC2, Fargate, Lambda | 1 or 3 years | Your stable compute baseline while the fleet still changes shape |
| **EC2 Instance Savings Plan** | up to ~72% (approximate, verify current) | Low — locked to one family in one region (size/OS still float) | 1 or 3 years | A baseline you are confident stays on the same family in the same region |
| **Standard Reserved Instance** | up to ~72% (approximate, verify current) | Low — locked to instance type; **sellable** in the RI Marketplace | 1 or 3 years | Non-compute services (RDS, ElastiCache, Redshift, OpenSearch) and estates already standardized on RIs |
| **Convertible Reserved Instance** | up to ~66% (approximate, verify current) | Medium — exchangeable for other types, but **not** sellable | 1 or 3 years | Largely superseded by Compute Savings Plans; only if you already live in the RI model |

Two properties that the discount column hides but that decide real cases. First, **Savings Plans are not refundable or resellable.** Once you commit $10/hr for three years, you owe it whether or not you use it — there is no marketplace to unload it, unlike a Standard RI. That asymmetry is the single strongest argument for under-committing. Second, **Savings Plans provide no capacity reservation.** They lower your rate; they do not guarantee the instance is available in your AZ when you need it. If guaranteed capacity is the actual requirement, that is a zonal Reserved Instance or an On-Demand Capacity Reservation, not a Savings Plan.

## Compute vs. EC2 Instance Savings Plans — which flexibility do you buy?

Take the Compute Savings Plan unless a baseline is genuinely locked to one instance family in one region. The Compute plan floats across family, size, region, and OS, and stretches across EC2, Fargate, and Lambda, for a few points less discount. The EC2 Instance plan buys those few points back by nailing you to, say, the `m` family in `us-east-1` for the whole term.

For a team that is still rightsizing — and every SMB reading this is still rightsizing — the flexibility is worth more than the points. You will move a service from `m6i` to `m7g` (Graviton) for a 20% saving, you will shift a workload from `us-east-1` to a cheaper region, you will re-platform something onto Fargate. Under a Compute Savings Plan, every one of those changes stays covered automatically. Under an EC2 Instance plan or a Standard RI, each one either strands the commitment or forces an exchange. Reserve the EC2 Instance plan for the settled, boring core of the estate whose shape you can defend a year out.

## Standard vs. Convertible Reserved Instances — what's the difference?

A Standard RI is locked to an instance type for the term at the highest RI discount, and can be sold in the RI Marketplace; a Convertible RI can be exchanged for a different type during the term at a lower discount, and cannot be sold. Convertible RIs existed to solve the "I want a commitment but my instance types keep changing" problem — which is exactly the problem Compute Savings Plans now solve, more cleanly and with broader reach. There is very little reason to write a new Convertible RI in 2026. If you want flexible commitment, buy a Compute Savings Plan. If you want maximum discount on something that will never change and might need selling later, buy a Standard RI. The middle option has been eaten.

## Should you still buy Reserved Instances in 2026?

Mostly not for EC2, but yes for two specific cases. For general EC2 compute, a Compute Savings Plan gives you the same discount ballpark with dramatically more flexibility, so new EC2 commitments should default to Savings Plans. Reserved Instances still earn their place in two situations:

1. **Services without a Savings Plan.** RDS, ElastiCache, Redshift, and OpenSearch are covered by Reserved Instances or reserved nodes, not by compute Savings Plans. The steady-state of your managed databases and caches is prime RI territory, and the sister guide on [the 30-day cost cleanup](/library/30-day-cloud-cost-cleanup) commits RDS via one-year, no-upfront RIs for exactly this reason. (SageMaker has its own Savings Plans; the point stands that coverage is per-service, so verify current before you assume.)
2. **Guaranteed capacity.** When you need a specific instance *available* in a specific AZ — not just discounted — a zonal Standard RI carries a capacity reservation. A Savings Plan does not.

Everything else that used to be an RI decision is now a Savings Plan decision.

## What coverage ratio should you target, and for how long?

Rightsize first, then cover about 70-80% of the stable baseline on a 1-year term unless the workload is genuinely durable. The order is not negotiable, because a commitment is a discount on *whatever you are running* — and if you are running an oversized fleet, you have just locked in the oversizing at a discount for the length of the term.

**Rightsize before you commit.** Run the [rightsizing pass that shrinks instances without breaking production](/library/rightsizing-aws-without-breaking-production) first, and clear out idle and orphaned resources, so the number you commit against reflects the workload you actually need — not the one that accreted over two years. Committing before rightsizing is the most common and most expensive sequencing error in FinOps: it converts a fixable overspend into a contractual one.

**Cover the stable baseline, not the peak.** Split your usage into the flat floor that is always on and the spiky part that comes and goes. Commit to roughly 70-80% of the floor. The uncovered slice absorbs growth, migration, and the services you might retire, and interruptible batch belongs on Spot rather than under any commitment at all. Aiming for 100% coverage feels thrifty and is a trap: the first rightsizing or decommission strands an unsellable, unrefundable hourly commitment.

**Know what "stable" means before you sign.** You can only commit confidently to a baseline you can actually see, which means committed spend has to be attributable to a team and a service. If your allocation runs on [cost-aware tags that survive reorgs](/library/cost-aware-tagging), someone owns each commitment and its renewal; if it does not, you are committing against a number nobody trusts. Pull the AWS Cost Explorer Savings Plans recommendations, but sanity-check them against your own baseline rather than accepting the number the engine that profits from commitment hands you.

**Term and payment.** Default to 1-year when there is any uncertainty; take 3-year only for workloads whose shape you can defend a year from now. Partial-upfront is the usual balance — most of the all-upfront discount without draining the cash — though a regulated shop watching runway should check the delta before parking capital with a vendor for three years.

## Which should you pick?

Pick On-Demand for anything spiky or uncertain, a Compute Savings Plan for the stable compute baseline, and a locked option (EC2 Instance SP or Standard RI) only for the durable core you would bet on a year out. The verdict table:

| Your situation | The call |
|---|---|
| Spiky, seasonal, or uncertain usage; a workload you might kill next quarter | On-Demand (Spot for interruptible batch) |
| A stable compute baseline, but the fleet still shifts families/sizes | Compute Savings Plan, 1-year |
| A large, genuinely durable baseline on a settled family + region | EC2 Instance SP or Standard RI, 3-year |
| RDS, ElastiCache, Redshift, or OpenSearch steady-state | Reserved Instances / reserved nodes for that service |
| You need guaranteed capacity in a specific AZ | Zonal Standard RI or On-Demand Capacity Reservation |
| Mid-migration, instance types still moving | Compute Savings Plan, 1-year (or stay On-Demand until it settles) |

The good-enough bar: a regulated SMB has made the right commitment call when its stable baseline is discounted, its spiky usage is not committed, and no single commitment covers usage that no longer exists. You do not need a FinOps platform or a 95% coverage trophy to clear that bar — you need to rightsize first, cover 70-80% of the floor with Compute Savings Plans, keep Reserved Instances for the non-compute steady-state, and prefer 1-year terms until a workload has earned three years of trust. The only genuinely wrong move is the one that feels most decisive: signing a big three-year deal to look aggressive about cost, before anyone has rightsized the thing you are about to reserve.

## The Template

Work top to bottom. The first section gates the rest — if you have not rightsized, do not buy anything yet.

**Before you commit (rightsize first)**
- [ ] Rightsizing pass complete; over-provisioned instances downsized and idle/orphaned resources deleted
- [ ] Stable baseline separated from spiky/seasonal usage in Cost Explorer
- [ ] Committed spend is attributable to a team/service via tags, so someone owns each renewal
- [ ] You are committing against the workload you need, not the one that accreted

**Sizing the commitment**
- [ ] Coverage target set at ~70-80% of the stable baseline, not 100%
- [ ] Spiky/uncertain usage left on On-Demand; interruptible batch moved to Spot
- [ ] Cost Explorer Savings Plans recommendations pulled and sanity-checked against your own baseline number
- [ ] You have accepted that Savings Plans are non-refundable and non-resellable before sizing them

**Choosing the instrument**
- [ ] Compute Savings Plan chosen for compute unless a baseline is truly locked to one family+region
- [ ] EC2 Instance SP / Standard RI reserved only for the settled, durable portion
- [ ] Reserved Instances (or reserved nodes) bought for non-compute steady-state (RDS, ElastiCache, Redshift, OpenSearch)
- [ ] Convertible RIs avoided for new commitments (Compute SP replaces them)
- [ ] Capacity actually guaranteed (zonal RI / Capacity Reservation) where availability, not just price, is the requirement

**Term and payment**
- [ ] 1-year term when there is any uncertainty; 3-year only for genuinely durable workloads
- [ ] Partial-upfront chosen as the default; all-upfront only when the discount delta justifies the cash
- [ ] Discounts verified against current AWS pricing, not this article's approximate figures

**After you commit**
- [ ] Savings Plans / RI utilization + coverage dashboards reviewed monthly
- [ ] Renewal dates calendared so nothing lapses silently back to On-Demand
- [ ] Commitment review folded into the quarterly cost-cleanup cadence

Most teams find they have been either over-committed on the wrong instrument or entirely uncommitted and paying full On-Demand. Both are normal starting points. The path out is the same: rightsize, then commit to the durable 70-80% with the most flexible instrument that fits, and revisit it every quarter.

## Frequently asked questions

**Q: Savings Plans vs Reserved Instances — which is better?**

For compute — EC2, Fargate, and Lambda — Savings Plans are better for most regulated SMBs, because you commit to a dollars-per-hour spend rather than to specific instances and keep flexibility across instance family, size, region, and operating system. AWS itself now steers most teams to Savings Plans. Reserved Instances still belong in two places: the non-compute services Savings Plans do not cover (RDS, ElastiCache, Redshift, OpenSearch), and zonal capacity reservations, since a Savings Plan is a discount, not a guarantee of capacity in an Availability Zone.

**Q: Should I buy Reserved Instances in 2026?**

For EC2 compute, usually no: a Compute Savings Plan gives a comparable discount with far more flexibility, and Convertible RIs are effectively superseded by it. Still buy Reserved Instances (or reserved nodes) for the steady-state of services that have no Savings Plan — RDS, ElastiCache, Redshift, OpenSearch — and use a zonal Standard RI or an On-Demand Capacity Reservation when you actually need guaranteed capacity in an Availability Zone. Verify current service coverage before you assume, since AWS changes what Savings Plans apply to over time.

**Q: What coverage ratio should I target?**

Target roughly 70-80% coverage of your stable baseline, not 100%. The uncovered 20-30% stays on On-Demand as a buffer for growth, migration, and the workloads you might turn off, and interruptible batch belongs on Spot rather than under a commitment. Commit to 100% and the first time you rightsize or retire a service you are paying an hourly commitment against usage that no longer exists — and unlike a Standard RI, you cannot sell a Savings Plan to recover it.

**Q: 1-year vs 3-year commitment?**

Default to 1-year when there is any uncertainty about the workload, the instance family, or the roadmap; the discount is smaller but you are not married to a decision for three years. Reserve 3-year terms for genuinely durable workloads whose shape you can defend a year from now. Partial-upfront is the common balance: most of the discount of all-upfront without draining the cash, though a real regulated shop should still check the delta before parking capital with a cloud provider for three years.

**Q: Compute vs EC2 Instance Savings Plans?**

A Compute Savings Plan floats across instance family, size, region, and operating system, and applies to EC2, Fargate, and Lambda, at a slightly lower discount. An EC2 Instance Savings Plan locks you to a single instance family in a single region (size and OS still float) in exchange for a higher discount. Most SMBs should take the Compute Savings Plan: the flexibility is worth the few points, because you will rightsize and shift families before the term ends. Only reach for the EC2 Instance plan on a baseline you are confident stays on the same family in the same region.

**Q: What's the difference between Standard and Convertible RIs?**

A Standard Reserved Instance is locked to an instance type/family for the term but carries the highest RI discount and can be sold in the AWS Reserved Instance Marketplace if you no longer need it. A Convertible Reserved Instance can be exchanged for a different configuration during the term but earns a lower discount and cannot be sold. In practice, Compute Savings Plans now do what Convertible RIs were for — flexible commitment — only better, so new commitments rarely have a reason to be Convertible RIs.

Template: "Commitment decision checklist" — branded PDF at https://kaansystems.com/templates/savings-plans-vs-reserved-instances.pdf

Source: https://kaansystems.com/library/savings-plans-vs-reserved-instances · Work with Kaan Systems: https://kaansystems.com/contact

---

# SLOs for Small Teams: Error Budgets Without a Platform Team

Canonical: https://kaansystems.com/library/slos-for-small-teams
Topic: Platform · Published: 2026-09-16 · Updated: 2026-10-12
Summary: An SLO is a target for a measured indicator over a window, and the error budget — 1 minus the SLO — is the failure you are allowed to spend. Small teams don't need an SRE function: pick one to three SLOs on your most user-facing service, measure from the user's side, and use the budget to gate risk. This is the concept the on-call and GitOps guides assume but never define.

## Key takeaways

- An SLO is a target for a measured indicator (an SLI) over a window — e.g. 99.9% of requests succeed and p95 latency under 300ms over 28 rolling days. The error budget is 1 minus the SLO: the failure you are allowed to spend.
- Small teams don't need an SRE function to run SLOs. Pick one to three SLOs on your single most user-facing service and ignore everything else until those earn their keep.
- Measure from the user's perspective — at the load balancer or with a synthetic check — never host CPU or the uptime of one box. CPU% and host uptime are vanity SLIs that stay green while users get errors.
- Use the error budget to gate risk: burning it fast freezes risky deploys, a healthy budget means ship faster. This is exactly the SLO-gated canary logic a GitOps rollout already runs, applied at human speed.
- Alert on burn rate, not raw thresholds. A page should mean "you are spending the month's budget too fast to make it," which is far quieter — and far more honest — than paging on every dip.

**SLOs for small teams** are not a shrunken version of Google's SRE book — they are one to three targets on your single most user-facing service, measured from the user's side, with an error budget you spend to decide when to slow down. You do not need an SRE function, a platform team, or a dashboard wall. You need a couple of numbers your team actually acts on. That is the entire discipline, and a 10-to-80-engineer regulated team can stand it up in an afternoon.

Two of the guides in this library lean on SLOs without stopping to define them. [On-call without burnout](/library/on-call-without-burnout) tells you to page only on symptoms that are user-facing and actionable — an SLO is how you decide what "user-facing" objectively means. [GitOps and progressive delivery](/library/gitops-progressive-delivery) gates every canary on an SLO check and rolls back automatically when it fails — that is an SLO doing its job at machine speed. This article is the missing definition underneath both, sized for a team with no SRE title on the org chart.

## What are SLOs, and do small teams need them?

Yes — a light version, and the concept matters more than the machinery. Three terms, in order:

- **SLI (Service Level Indicator):** something you measure about the service from the user's perspective. The share of requests that succeed. The p95 request latency. A measured number, nothing more.
- **SLO (Service Level Objective):** a target for that indicator over a window. "99.9% of requests succeed, and p95 latency stays under 300ms, over 28 rolling days." The SLI is the number; the SLO is the line you draw across it.
- **Error budget:** 1 minus the SLO. At a 99.9% target, 0.1% of requests are *allowed* to fail. That allowance is not a failure — it is a resource you get to spend on risk.

Small teams need the concept, not the organization. The two failure modes are equal and opposite. Skip SLOs entirely and on-call pages on noise, because nobody ever defined what "broken" means, so every blip is a judgment call at 2 a.m. Import the full enterprise SRE apparatus and you get an SLO for every microservice, a wall of dashboards, and a ritual nobody consults. The middle — one to three SLOs on the service that matters, wired into how you deploy and how you page — is the whole win, and it is reachable this quarter.

## SLO vs SLA vs SLI — what's the difference?

An SLI is what you measure, an SLO is the internal target you hold yourself to, and an SLA is the contractual promise you make a customer. They are frequently blurred, and the blur is expensive.

| Term | What it is | Who it's for | Teeth |
|---|---|---|---|
| **SLI** | A measured indicator (success rate, latency) | Your engineers | None — it's just a number |
| **SLO** | Internal target for an SLI over a window | Your team | Error budget; gates deploys and on-call priorities |
| **SLA** | Contractual promise to a customer | Customers, sales, legal | Financial penalties for a miss |

The operative rule: set the SLO stricter than any SLA you sign. If you promise a customer 99.5%, run an internal SLO of 99.9%, so your error budget burns and warns you well before the contract is at risk. Most regulated SMBs have SLOs long before they have a signed SLA — and that is the correct order. Get the internal target working first; the contractual number is a business decision layered on top of a system you already understand.

## Which SLIs and SLOs should a small team start with?

Start with one to three, all on your single most user-facing service, all measured from the user's side. Here is the starter menu. Take the first row as non-negotiable, then add at most two more.

| SLI (measured at the edge) | Starter SLO (28-day rolling) | Why this one |
|---|---|---|
| **Request success rate** — non-5xx share of responses at the load balancer | 99.9% of requests succeed | The core "is it working for users" signal. Measured at the LB, not inside the app, so it catches the app being entirely down — the case an in-process metric never reports because the process isn't running to report it. |
| **Latency** — p95 request duration at the edge | p95 < 300ms | "Up but slow" is still broken. p95 catches the tail users actually feel without chasing p99.9 noise on a low-traffic service. |
| **Critical-journey availability** — synthetic check of one real flow (login, checkout) | 99.9% of synthetic runs pass | Catches whole-path failures a per-request rate misses: auth broken, a dependency down, a page that returns 200 but renders nothing. |
| **Freshness** — a business-critical async job (export, billing, ETL) completes on time | 99% within its expected window | The one background pipeline whose lateness a customer or an auditor actually notices. Skip this row if you have no such job. |

The first row earns its place on every service. Add latency if users wait on synchronous responses. Add the synthetic check if a login or checkout path *is* the product. Then stop — three is plenty for a team this size.

What does **not** belong in this table is as important as what does. Host CPU%, memory pressure, requests per second, and the uptime of a single box are diagnostics, not indicators of user pain. A box can sit at 90% CPU while every user is perfectly happy, and idle at 10% while the service returns 500s to everyone. These are vanity SLIs: they generate green dashboards and false confidence. Keep them for debugging an incident; never make them an SLO.

## What's a good availability target?

99.9% is the right default for almost every SMB service. Three nines is achievable without a follow-the-sun rotation, and it buys a downtime budget your team can actually operate inside. The number that matters is not the count of nines — it is the failure the target permits over your window:

| SLO | Allowed failure (28-day window) |
|---|---|
| 99% | ~6.7 hours |
| 99.9% | ~40 minutes |
| 99.95% | ~20 minutes |
| 99.99% | ~4 minutes |

Look at the 99.99% row and ask, honestly, whether a 10-to-80-engineer team can detect *and* fix an incident inside four minutes without staffing around the clock. You can't, so don't promise it — a target you can't meet is worse than an honest one, because it burns credibility every month. Resist the opposite temptation too: 100% is not a target, it's a way to freeze all shipping in the name of a number you'll never hit. Set availability where your team can genuinely respond, and keep it consistent with your [on-call rotation](/library/on-call-without-burnout) — the SLO defines what the pager is allowed to wake someone for, so the two have to agree.

## How do error budgets actually work?

The error budget is 1 minus the SLO, and the point is that you *spend* it. A healthy budget is permission to take risks — ship faster, run the schema migration, deploy on a Friday. A budget you're burning through says the reverse: freeze risky changes and spend what's left on stability until it recovers. Two rules, written down, turn the abstraction into behavior:

- **Budget healthy** (say, more than half remaining over the 28-day window): normal velocity. Canaries auto-promote, migrations proceed, the team ships.
- **Budget burning fast or exhausted:** freeze non-essential deploys, no risky migrations, and on-call's job shifts from features to reliability until the budget climbs back.

This is not theoretical for anyone running GitOps. It is exactly the [SLO-gated canary](/library/gitops-progressive-delivery) logic already in a progressive rollout: every deploy shifts a slice of traffic, runs an analysis against the SLO, and aborts itself when the new version would burn budget too fast. The error budget is that same rule applied at human speed — the machine enforces it per deploy, the team enforces it per week.

The other half is how you alert. The mistake is paging on the raw SLI — "success rate dipped below 99.9% for one minute." That fires constantly, and a team trained to ignore a noisy alert will ignore the one that matters, which is the precise trap [on-call without burnout](/library/on-call-without-burnout) is written to avoid. Alert on **burn rate** instead: how fast you are consuming the window's budget. A page then means "at this rate you'll blow the month's budget before it resets" — rare, real, and worth waking up for. A brief blip that barely dents the budget is a line on a dashboard, not a 2 a.m. wakeup.

## How do you measure an SLO without an SRE team?

You already have the data — you need the load balancer's metrics and one synthetic check, not a platform hire. Three practical rules:

- **Measure at the edge.** Your ALB, NLB, or ingress already emits per-request status and latency. That is your success-rate and p95 SLI for free, and critically it's from the *user's* side of the application, which is the entire point. Host-level metrics measure the wrong thing from the wrong vantage.
- **Add one synthetic check.** A scheduled probe that runs a real user journey end-to-end catches the failures request metrics miss — the login that succeeds but the dashboard renders blank. Any uptime or synthetic-monitoring tool does this; you don't build it, you configure it.
- **Use a 28-day rolling window.** Not a calendar month (the reset hides problems that straddle the boundary), not 7 days (too twitchy to trust). Twenty-eight days is four weeks of the same weekday mix, and because it rolls, the budget is always "the last four weeks" — which is how humans actually reason about whether things have been healthy lately.

This all assumes the observability floor exists: request metrics scraped and stored somewhere you can query them. That floor is part of [what "production-ready" actually means for a cluster](/library/production-ready-kubernetes), and standing up SLOs is one of the reasons a real [platform engineering foundation](/library/platform-engineering-for-regulated-startups) exists in the first place. If you don't have request metrics yet, that's the prerequisite — and it's cheap, not a project.

## The good-enough bar

A small team has good-enough SLOs when three things are true: one to three SLOs exist on the most user-facing service, all measured from the user's side; a 28-day error budget is defined and visible; and there is a *written rule* for what happens when the budget burns — deploys slow, on-call priorities shift. That's the bar. No dashboard wall, no SLO for every service, no SRE title required.

The wrong version is either extreme. No SLOs means every alert is a guess and every "are we down?" is a debate. An SLO spreadsheet nobody opens is theater. The honest test is behavioral: if a burning error budget doesn't change a single decision — doesn't freeze a deploy, doesn't reprioritize on-call — then you don't have an SLO, you have a chart. Delete it and start over with one number the team will act on.

## The Template

A starter SLO worksheet. Fill one block per SLO; a small team has one to three of these total, not one per service. Publish it where on-call can find it, and revisit it quarterly.

**One block per SLO (aim for 1-3 total)**
- Service: ________________ (your single most user-facing service first)
- SLI — what you measure, from the user's side: ________________
- Measured where: ☐ load balancer / ingress ☐ synthetic check ☐ ____________ (not host CPU, not RPS)
- Target (SLO): ______% over a 28-day rolling window (and/or p95 < ______ ms)
- Error budget: 100% − SLO = ______ (e.g. 99.9% → 0.1% ≈ 40 min / 28 days)
- Burn-rate alert fires when: on track to exhaust the budget before the window resets — NOT on every dip
- When the budget is HEALTHY (> ____% left): ☐ ship normally ☐ canaries auto-promote ☐ migrations proceed
- When the budget is BURNING / EXHAUSTED: ☐ freeze risky deploys ☐ no migrations ☐ on-call pivots to reliability
- Owner — who watches this number: ________________
- Reviewed quarterly: ☐

Two rules that keep the worksheet alive rather than decorative:

- **Start with one.** The most user-facing service, one success-rate SLO, one error budget. Add the second only after the first has changed a real decision. A single working SLO beats four aspirational ones.
- **If it never changes a decision, delete it.** An SLO that never freezes a deploy or greenlights a risk is not measuring anything you use. Cut it, and put the attention on the number that does.

Get one SLO to the point where a burning budget actually slows the team down and a healthy one speeds it up, and you have the whole discipline — the same loop the [GitOps rollout](/library/gitops-progressive-delivery) runs automatically and the [on-call system](/library/on-call-without-burnout) leans on, minus the platform team you were told you needed.

## Frequently asked questions

**Q: What are SLOs, and do small teams need them?**

An SLO (Service Level Objective) is a target for a measured indicator of health over a window — for example, 99.9% of requests succeed and p95 latency stays under 300ms over 28 rolling days. Yes, small teams need them, but a light version: one to three SLOs on your most user-facing service, not the full enterprise SRE apparatus. Skipping SLOs entirely is why on-call pages on noise and nobody can say whether the service is actually healthy; importing all of Google's SRE book is a dashboard wall nobody reads. The middle — a couple of SLOs your team acts on — is the whole win.

**Q: SLO vs SLA vs SLI — what's the difference?**

An SLI (Service Level Indicator) is what you measure — success rate, latency. An SLO (Service Level Objective) is the internal target you hold that indicator to over a window. An SLA (Service Level Agreement) is a contractual promise to a customer, usually with financial penalties for missing it. The rule: set the SLO stricter than any SLA, so your error budget burns and warns you before the contract does. Most SMBs have SLOs long before they sign an SLA, and that is the correct order.

**Q: How many SLOs should a small team start with?**

One to three, all on your single most user-facing service. Start with request success rate. Add p95 latency if users interact synchronously, and add a synthetic check of one critical journey (login, checkout) if that path is your product. Then stop. An SLO for every service is how a small team ends up with a spreadsheet nobody consults; a few SLOs on the service that matters is how you get a number people actually act on.

**Q: What's a good availability target?**

99.9% (three nines) is the right default for almost every SMB service: achievable without heroics, and it buys a downtime budget of roughly 40 minutes over a 28-day window that a small team can actually operate inside. Avoid both extremes — 100% is impossible and chasing it just freezes shipping, and five nines (about 4 minutes per 28 days) is an enterprise cost you are years from needing. Set the target where your team can realistically detect and fix an incident, so the SLO and your on-call rotation are consistent with each other.

**Q: How do error budgets work?**

The error budget is 1 minus the SLO — at 99.9%, one request in a thousand is allowed to fail — and you spend it deliberately. A healthy budget is permission to take risks: ship faster, run the migration, deploy on a Friday. A budget you're burning through means the opposite: freeze risky changes and spend the remaining budget on stability until it recovers. Write two rules down — what you do when the budget is healthy, and what you do when it's burning — and the abstract number becomes an actual decision.

**Q: How do you measure an SLO without an SRE team?**

You already have the data. Your load balancer or ingress emits per-request status and latency, which gives you a success-rate and p95 SLI from the user's side of the app for free — no host agents, no platform team. Add one synthetic check that exercises a real user journey end-to-end to catch whole-path failures request metrics miss. Score both over a 28-day rolling window. The only prerequisite is having request metrics somewhere you can query them, which is part of the observability floor a production-ready cluster already needs.

Template: "Starter SLO worksheet" — branded PDF at https://kaansystems.com/templates/slos-for-small-teams.pdf

Source: https://kaansystems.com/library/slos-for-small-teams · Work with Kaan Systems: https://kaansystems.com/contact

---

# SOC 2 for Startups: The Complete Readiness Guide

Canonical: https://kaansystems.com/library/soc2-for-startups-complete-guide
Topic: Compliance · Published: 2026-09-16 · Updated: 2026-10-12
Summary: Most startups pursue SOC 2 because a deal requires it, then discover it's less a certificate than a standing operational commitment. This is the map: what SOC 2 is, Type I vs Type II, which Trust Services Criteria to scope, how long it takes, what it costs, and the engineering work your team actually has to build — with links to the deep dives for each step.

## Key takeaways

- SOC 2 is a CPA-issued attestation that your controls meet the AICPA Trust Services Criteria — for startups it's almost always driven by an enterprise customer's security review, not a legal requirement.
- Type I attests your controls are designed correctly at a point in time; Type II attests they operated effectively over a 3-12 month window. Buyers want Type II; Type I is a fast interim signal.
- Only the Security criterion is mandatory. Scope the other four (Availability, Confidentiality, Processing Integrity, Privacy) to what you actually commit to — over-scoping is the most common early mistake.
- Budget realistically: auditor fee, a compliance-automation platform, a penetration test, and — the biggest hidden cost — the engineering time to build and evidence the controls.
- A compliance platform automates the evidence, not the controls. You still have to consolidate identity, centralize logging, enforce change management, and manage vendor risk.
- Treat SOC 2 as an operating posture, not a document exercise: the report expires, the observation window is continuous, and manual evidence collection is what makes renewals painful.

**What is SOC 2 for a startup, and how do you actually get it?** SOC 2 is an audit report — issued by a licensed CPA firm — attesting that your security controls meet the AICPA's Trust Services Criteria. For a startup it's almost always a *sales* requirement, not a legal one: an enterprise buyer's security review demands it before they'll sign. Getting it means scoping the right criteria, building real controls (access, logging, change management, vendor risk), running them for an observation window, and collecting the evidence an auditor will believe. This guide is the whole path, with links to the deep dives for each step.

This is the hub for our compliance library. Each section here is the map; the linked articles are the terrain. If you're still deciding whether a subscription is the work, start with [evidence pipelines](/library/evidence-pipelines-101): the audit asks for controls in operation, and Terraform plus CI is where that evidence already lives.

## Do you actually need SOC 2 yet?

Usually you pursue SOC 2 the quarter a real deal stalls on it — not before. It's expensive and it's a standing commitment, so doing it speculatively burns money and attention you need elsewhere. The honest trigger is simple: a customer you want has put "provide a SOC 2 report" in their procurement path, or you're losing deals in security review. When that happens, SOC 2 stops being overhead and becomes revenue you're leaving on the table.

The report also pays a secondary dividend: it collapses the security-questionnaire tax. Instead of hand-answering a 300-line spreadsheet per prospect, you hand over the report and a few artifacts — see [reading a security questionnaire](/library/reading-security-questionnaire) for how to shortcut that process even before the report lands.

## What's actually in scope: the five Trust Services Criteria

SOC 2 is built on five Trust Services Criteria (TSC). Only the first is mandatory; the rest are scoped to what you commit to.

| Criterion | Mandatory? | What it covers | Scope it when… |
|---|---|---|---|
| **Security** (Common Criteria) | **Yes** | Access control, change management, risk management, monitoring — the baseline | Always |
| Availability | No | Uptime, DR, capacity, monitoring | You make uptime commitments (SLAs) |
| Confidentiality | No | Protecting data designated confidential | Contracts promise confidentiality beyond the norm |
| Processing Integrity | No | Processing is complete, valid, accurate, timely | You process transactions/data on customers' behalf |
| Privacy | No | Handling of personal information per your notice | You collect consumer PII and make privacy commitments |

Most startups scope **Security only** for the first report, occasionally adding Availability or Confidentiality when a contract explicitly requires them. Adding criteria multiplies the controls you must build and evidence — so scope to the commitments you've actually made, not to look thorough.

## Type I vs. Type II: which do you start with?

Type I proves the controls are *designed* right on one day; Type II proves they *ran* over a period. The difference is the whole value.

| | Type I | Type II |
|---|---|---|
| What it attests | Controls suitably designed at a point in time | Controls operated effectively over a period |
| Observation window | None (a snapshot) | Typically 3-12 months |
| What buyers accept | A weak interim signal | The real thing |
| Time to obtain | Fastest (weeks) | The window + the audit |
| Best used as | An interim proof while you run the window | The report you actually hand customers |

The common startup path: reach **Type I** readiness fast to unblock a deal with an interim report, start the observation window the same day, and convert to **Type II** once the window closes. Some teams skip Type I entirely and go straight to a short (3-month) Type II window.

## The readiness timeline

The path is the same regardless of tooling: scope, find the gaps, fix them, then run and evidence the controls. Most of that remediation is platform work your Terraform already records. The [platform engineering guide](/library/platform-engineering-for-regulated-startups) is the map.

```mermaid
flowchart LR
  S["Scope<br/>criteria + systems"]
  G["Gap<br/>assessment"]
  R["Remediate<br/>build the controls"]
  T1["Type I<br/>point-in-time"]
  O["Observation<br/>window (3-12 mo)"]
  T2["Type II<br/>audit + report"]

  S --> G --> R --> T1 --> O --> T2

  classDef done fill:#1F4A3E,stroke:#3FA585,stroke-width:2px,color:#E8F5EE;
  classDef next fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
  class S,G,R done;
  class T1,O,T2 next;
```

The load-bearing phase is remediation — actually building the controls — and it's where engineering, not compliance, does the work.

## What it costs

Four line items, and the fourth is the one teams underestimate:

- **Auditor fee.** The CPA firm that issues the report. For a startup this is typically five figures and scales with scope and report type.
- **Compliance-automation platform.** Vanta, Drata, Secureframe, and peers automate evidence collection — a few thousand to low five figures per year. Worth it for most startups; see [do you need a compliance platform](/library/do-you-need-a-compliance-platform) for the honest build-vs-buy.
- **Penetration test.** Frequently expected by auditors and buyers; mid four to low five figures.
- **Engineering time.** The largest and least-visible cost — building and then continuously evidencing the controls. Budget it explicitly or it comes out of your roadmap by surprise.

## The engineering work (what your team actually builds)

This is the part a platform can't do for you. The controls that satisfy the Security criterion are, overwhelmingly, ordinary good platform engineering:

- **Access & identity.** SSO everywhere, MFA on every account, least-privilege roles, and one-action offboarding. The single highest-leverage move is [killing long-lived credentials](/library/killing-long-lived-credentials) in favor of federated, short-lived access — auditors and attackers care about the same thing here.
- **Change management.** Every production change flows through a reviewed pull request and a gated pipeline, with a traceable history. That history *is* your change-management evidence.
- **Logging & monitoring.** Centralized, tamper-resistant logs with alerting. The same discipline behind [securing PHI in observability](/library/securing-phi-observability) applies to SOC 2 — protect the logs, and make them prove what happened.
- **Encryption.** At rest and in transit, with sane key management. If you're also on the healthcare path, the [minimum-viable HIPAA on AWS](/library/minimum-viable-hipaa-aws) controls overlap heavily with SOC 2's.
- **Vendor / subprocessor risk.** A maintained subprocessor list with signed DPAs and a review cadence — [vendor risk scoring](/library/vendor-risk-scoring) is the lightweight method.
- **Evidence collection.** The ongoing burden: proving, continuously, that the above operate. Automate it or drown in screenshots. [Evidence pipelines 101](/library/evidence-pipelines-101) is the pattern that turns evidence from a quarterly fire drill into a byproduct of running the system.

## SOC 2 vs. HIPAA vs. HITRUST

SOC 2 isn't the only framework, and picking the wrong one wastes a year. In short: SOC 2 is a flexible, sales-driven *attestation*; HIPAA is a US legal requirement for handling PHI (not a certification you "pass"); HITRUST is a prescriptive, certifiable framework often demanded in healthcare. Which one you need depends on who you sell to and what data you touch — the full decision is in [choosing between SOC 2, HITRUST, and HIPAA](/library/choosing-soc2-hitrust-hipaa).

## Common pitfalls

- **Over-scoping the criteria.** Adding Availability/Privacy "to be safe" multiplies work for no sales benefit.
- **Treating it as a document exercise.** A binder of policies with no operating controls fails a Type II and, worse, doesn't make you secure.
- **Manual evidence.** If evidence collection is a person taking screenshots, every audit and renewal is a fire drill.
- **Buying the platform before consolidating identity.** The platform evidences controls that don't exist yet. Fix identity first.
- **No owner.** SOC 2 without a named owner drifts; the observation window quietly breaks and nobody notices until the audit.

## What does your Terraform already prove?

A large share of the Security criterion is platform state, if the four jobs in the [platform engineering guide](/library/platform-engineering-for-regulated-startups) are in place. The account boundary is the isolation control. The git log is the change-management control. Short-lived credentials are the access control. CloudTrail, CI, and Terraform state are the evidence, which is what [evidence pipelines](/library/evidence-pipelines-101) collect. The report is the attestation. The engineering is the platform.

## The Template

A SOC 2 readiness roadmap. Work top to bottom; each phase makes the next meaningful. Score each item Not started / In progress / Done.

**Phase 1 — Scope**
- [ ] Confirmed the business trigger (a real deal / lost deals in security review)
- [ ] Chosen Type I-then-II vs. straight-to-Type-II
- [ ] Scoped criteria (Security + only the others you commit to)
- [ ] Defined the systems/boundaries in scope

**Phase 2 — Gap assessment**
- [ ] Mapped current controls to the Common Criteria
- [ ] Identified the gaps (access, change mgmt, logging, encryption, vendor risk, evidence)
- [ ] Named an owner for the program

**Phase 3 — Remediate (the engineering work)**
- [ ] SSO + universal MFA; least-privilege roles; one-action offboarding
- [ ] Long-lived credentials eliminated in favor of federated, short-lived access
- [ ] Change management: reviewed PRs + gated pipeline with traceable history
- [ ] Centralized, tamper-resistant logging + alerting
- [ ] Encryption at rest + in transit with managed keys
- [ ] Subprocessor list + signed DPAs + review cadence
- [ ] Backup/DR + a tested restore
- [ ] Incident-response plan documented and exercised

**Phase 4 — Evidence + observe**
- [ ] Compliance platform integrated (or an evidence pipeline stood up)
- [ ] Evidence collection automated, not manual
- [ ] Observation window started; controls running continuously

**Phase 5 — Audit**
- [ ] Auditor engaged; penetration test complete
- [ ] Type I issued (if used as interim)
- [ ] Type II window closed; report issued and shareable with prospects

Most startups find Phase 3 is 80% of the effort and Phase 4 is where discipline pays off or falls apart. Both are engineering problems more than compliance ones.

## Frequently asked questions

**Q: What is SOC 2 for a startup?**

SOC 2 is an audit report, issued by a licensed CPA firm, attesting that your company's security controls meet the AICPA's Trust Services Criteria. For a startup it is almost never a legal requirement — it's a sales unlock: enterprise and mid-market buyers require it in their vendor security reviews before they'll sign. You pursue it to remove a purchasing blocker, and the report becomes proof you can hand a prospect instead of answering a 300-line questionnaire from scratch.

**Q: What's the difference between SOC 2 Type I and Type II?**

Type I attests that your controls are designed appropriately at a single point in time; Type II attests that those controls actually operated effectively over a period, typically 3 to 12 months. Buyers almost always want Type II because design without operation proves little. Startups often get a Type I first as a fast interim signal, then run the observation window and follow with Type II.

**Q: How long does SOC 2 take for a startup?**

A focused startup can reach Type I readiness in roughly 6 to 10 weeks — scoping, a gap assessment, and remediation of the missing controls. Type II then requires an observation window (3 months is the common minimum, 6-12 for a mature report) during which the controls must run continuously, followed by the audit itself. So the realistic timeline from start to a Type II report is a quarter to two quarters, most of it the observation period.

**Q: How much does SOC 2 cost?**

For a startup, plan for four line items: the auditor's fee (often five figures), a compliance-automation platform (a few thousand to low five figures per year), a penetration test (often required, mid four to low five figures), and the engineering time to build and evidence the controls — the largest and most-underestimated cost. First-year totals for a small startup commonly land in the low-to-mid five figures plus significant internal effort.

**Q: Which Trust Services Criteria do I need for SOC 2?**

Only Security (the Common Criteria) is mandatory. The other four — Availability, Confidentiality, Processing Integrity, and Privacy — are optional and should be scoped to the commitments you actually make to customers. Most startups scope Security alone, sometimes adding Availability or Confidentiality if their contracts promise them. Over-scoping the criteria is the most common early mistake and adds cost and evidence burden for no sales benefit.

**Q: Do I need a compliance platform like Vanta or Drata for SOC 2?**

You don't strictly need one, but for most startups it pays for itself by automating evidence collection — pulling configuration from AWS, GitHub, your identity provider, and so on, mapping it to the criteria, and monitoring continuously. What it does not do is build the controls: you still have to consolidate identity, centralize logging, enforce change management, and manage vendor risk. Buy the platform to automate the evidence, not to substitute for the engineering.

Template: "SOC 2 readiness roadmap" — branded PDF at https://kaansystems.com/templates/soc2-for-startups-complete-guide.pdf

Source: https://kaansystems.com/library/soc2-for-startups-complete-guide · Work with Kaan Systems: https://kaansystems.com/contact

---

# Vanta vs. Drata vs. Doing It Yourself: SOC 2 Automation Compared

Canonical: https://kaansystems.com/library/vanta-vs-drata-soc2
Topic: Compliance · Published: 2026-09-16 · Updated: 2026-10-12
Summary: Vanta and Drata are the two leading SOC 2 automation platforms, and teams agonize over the choice as if it were load-bearing. It mostly isn't. Both connect to your stack, auto-collect evidence, and monitor continuously; the real decision is integration coverage for your systems, the auditor relationship, and price — plus the honest DIY column most comparisons leave out.

## Key takeaways

- Vanta and Drata are the two leading SOC 2 automation platforms; the differences are marginal and shift release to release. Pick by integration coverage for YOUR stack, the auditor relationship, and price — not the brand on the dashboard.
- Both connect to your stack (AWS, GitHub, your IdP, HR), auto-collect evidence, map it to the Trust Services Criteria, and monitor continuously; both run auditor marketplaces. Secureframe, Thoropass, and Sprinto are credible alternatives worth a quote.
- The load-bearing fact: a compliance platform automates the EVIDENCE, not the CONTROLS. You still build access control, change management, centralized logging, and encryption yourself — the dashboard only observes whether they exist.
- DIY — a self-hosted evidence pipeline — wins only if you already have the engineering discipline to run one, and it decays badly past roughly 20 controls or the first Type II observation window.
- The good-enough bar: whichever tool evidences the most of YOUR controls automatically, sits on your chosen auditor's marketplace, and you can still afford at year-two renewal. For most regulated SMBs, both Vanta and Drata clear it.

**Vanta vs. Drata: which SOC 2 automation platform should you buy?** Whichever one has native connectors for the most of your exact stack, works with the auditor you want, and costs less at year-two renewal — because on everything else the two are close enough that the brand is not the decision. Both connect to your cloud, code, identity, and HR systems, auto-collect evidence, map it to the Trust Services Criteria, and monitor continuously. The differences are marginal and shift release to release. This guide compares them across the dimensions that actually differ, adds the DIY column most vendor bake-offs omit, and settles the one fact the sales process is built to blur: a platform automates the evidence, not the controls.

This is the tool-selection companion to two deeper pieces. If you are still deciding whether to buy any platform at all, start with [do you need a compliance platform](/library/do-you-need-a-compliance-platform); for the whole SOC 2 path — Type I vs. Type II, the five criteria, the timeline, and the cost — the [SOC 2 for startups complete guide](/library/soc2-for-startups-complete-guide) is the hub this article sits under.

## What do Vanta and Drata actually do?

The same thing, competently. Both are evidence-automation platforms: read-only connectors reach into your identity provider, your cloud accounts, your source host, your MDM, and your HR system, and pull compliance evidence automatically and continuously — MFA status per user, encryption settings per bucket, branch protection per repo, offboarding timestamps per departure. That evidence gets mapped to the SOC 2 Common Criteria (and to ISO 27001, HIPAA, or other frameworks if you scope them), tracked against control owners with deadlines, and surfaced on an auditor-facing dashboard so your CPA firm reads evidence in the tool instead of over email attachments.

That is the product, and it is genuinely valuable. Manual evidence collection for a SOC 2 cycle runs 50 to 100-plus hours of mostly senior engineer time, all of it miserable, and it has to be redone every cycle because auditors need evidence from the current period. A platform does that collection on a schedule, forever. Continuous monitoring is the second real benefit: a control that silently degrades in March surfaces in March, not during next January's fieldwork.

Vanta and Drata are the two market leaders, but they are not the only credible options — Secureframe, Thoropass, and Sprinto do the same job and are worth a quote, especially if one of them has better coverage for your particular stack. The category matters more than the logo.

## Vanta vs. Drata vs. DIY: how do they compare?

Close on capability, different on the details that touch your specific environment. The table is the comparison in one view. Treat every cell as directionally true and verify current specifics — feature parity and pricing in this category move quarter to quarter, and the vendors leapfrog each other on connectors constantly.

| Dimension | Vanta | Drata | DIY / no platform |
|---|---|---|---|
| Framework coverage (SOC 2, ISO 27001, HIPAA, GDPR) | Broad multi-framework; SOC 2, ISO 27001, HIPAA, GDPR, and more, with cross-framework control mapping | Broad multi-framework; the same major frameworks with cross-mapping | Whatever you build mappings for by hand; one framework is realistic, several is a project |
| Integration / evidence-automation breadth | Large native connector catalog across cloud, IdP, MDM, HR, code | Large native connector catalog, comparable in scope | Only what you wire yourself (CloudTrail, AWS Config, IdP + HR logs); anything unbuilt is a manual upload |
| Continuous control monitoring | Continuous; alerts on drift and failing controls | Continuous; alerts on drift and failing controls | Only if you build alerting on top of your pipeline |
| Auditor network | Marketplace of partner firms familiar with the evidence format | Marketplace of partner firms familiar with the evidence format | Bring your own auditor; they read your artifacts cold |
| Pricing model | Annual subscription, quoted per company; steps up by employee tier + framework count | Annual subscription, quoted per company; steps up by employee tier + framework count | No subscription; cash cost near zero, engineering time is the price |
| UX / onboarding | Guided onboarding, polished dashboard, policy template library | Guided onboarding, polished dashboard, policy template library | You own the whole experience; no guardrails, no templates |
| What it does NOT do | Implement any control | Implement any control | Implement any control |

The last row is the one that matters most, and it is identical across all three columns. Hold that thought.

## What does a compliance platform NOT do?

It does not implement a single control — and that distinction is the entire buying decision. A platform observes and reports; it does not fix. Connect Vanta or Drata to your AWS account and it will tell you, accurately and continuously, that three S3 buckets are public, that four IAM users have no MFA, and that CloudTrail is off in two regions. It will not turn on MFA. It will not separate your prod and dev accounts. It cannot. Those are engineering projects, and they were engineering projects before you started paying five figures a year for a dashboard that describes them.

This is the fact the sales funnel is structured to blur, so it is worth stating flatly: the red items on a compliance dashboard are your engineering backlog, priced in sprints, not a compliance to-do list a subscription clears. The controls underneath SOC 2 are ordinary good platform engineering — SSO everywhere and universal MFA, [killing long-lived credentials](/library/killing-long-lived-credentials) in favor of federated short-lived access, reviewed PRs through a gated pipeline, centralized tamper-resistant logging, encryption at rest and in transit, and a maintained subprocessor list. You build those. The platform watches whether they operate. If you buy the platform before the controls exist, the first year of the subscription is pure rent on a wall of red.

None of that is a knock on the products. It is their design: they are observation and workflow tools, and observation does not close gaps. Which is exactly why the platform-vs-platform question is less important than teams treat it — you are choosing between two competent cameras pointed at work only you can do.

## How much do Vanta and Drata cost?

Both land in roughly the low-to-mid five figures per year for SOC 2 at typical SMB headcount — but treat that as a directional, verify-current figure, because neither publishes firm public pricing and both quote per company. The number you see in year one is a land grab; year two is where the business model lives. Two step-up axes drive the renewal: employee tiers and framework count. Crossing 50 or 100 employees mid-contract, or adding ISO 27001 or HIPAA at list price, routinely turns a modest first year into a materially larger second one.

The defense is boring and effective: get the multi-framework add-on price and the employee-tier boundaries written into the order form before you sign, not left in the sales thread. Teams that negotiate the HIPAA or ISO add-on at initial signature routinely pay far less than teams that add it at renewal — by then your evidence history lives in the tool and the vendor knows what migration would cost you. And remember the subscription is never the whole bill: the auditor's fee and the engineering time to build the controls are separate line items on every path, platform or not.

## When is doing it yourself the right call?

When you already have the engineering discipline to run an evidence pipeline — and almost never otherwise. Structurally, Vanta and Drata are commercial [evidence pipelines](/library/evidence-pipelines-101): connectors in, durable queryable evidence out, mapped to controls. The self-hosted version is the same idea for less cash and more engineering time. A team already comfortable wiring CloudTrail, AWS Config, and IdP logs into S3 with Object Lock and a query layer can absolutely produce audit evidence an auditor will accept, on their own terms, with no subscription.

But be honest about the two cases where DIY actually wins. The first is a single, first Type I at tiny scope — one product, a couple of AWS accounts, a handful of people — where you accept 100-plus hours of manual screenshot collection because the budget is genuinely near zero. The second is the disciplined-pipeline team above. Outside those, DIY manual collection decays badly past about 20 controls, and a Type II observation window means re-collecting quarterly forever. If nobody owns that pipeline as a real, staffed responsibility, "we'll do it ourselves" becomes "we reconstruct evidence under deadline every cycle," which is the exact failure the platforms exist to kill. An [evidence pipeline](/library/evidence-pipelines-101) is what unstaffed DIY quietly fails to keep running, which is why a platform looks attractive once the controls exist.

## Which should you pick?

Pick by integration coverage, auditor, and price — in that order — and stop treating the brand as the decision. Walk it in three steps:

1. **Integration coverage for your exact stack.** Put both Vanta and Drata (and Secureframe, if you like) in a read-only trial for a week against your real systems. Count how many of your controls each evidences automatically versus how many become manual uploads. An unsupported IdP, a self-hosted GitLab, or a niche MDM turns "automated" back into the spreadsheet you were trying to escape — now with a subscription fee. This step usually produces a clear winner on its own.
2. **The auditor relationship.** If you already have a CPA firm you trust, confirm it works on the platform. If you do not, both marketplaces are fine starting points — check which partner firms completed audits recently and at what fee range.
3. **Price at renewal, not signature.** Get the year-two list price, the employee-tier boundaries, and the multi-framework add-on cost in writing before you commit.

**The good-enough bar:** you have made the right call when the tool you chose evidences the majority of your controls automatically, sits on the marketplace of an auditor you can work with, and stays affordable at year-two renewal. Both Vanta and Drata clear that bar for most regulated SMBs at 10 to 80 engineers — which is precisely why the agonizing between them is misspent energy. The failure mode is not picking the "wrong" platform; the two are close enough that either is defensible. The failure mode is buying either one before the controls exist, or buying it to avoid building them at all. Choose DIY only if a named engineer owns the evidence pipeline as real work; choose a platform — Vanta or Drata, it barely matters which — if you want the 50-to-100-hour-per-cycle collection burden handled and the controls are (or are about to be) real.

## The Template

Score each item honestly before you sign anything. This is a selection scorecard, not a checklist to complete — it tells you which column you are in.

**Step 1 — Are the controls actually in?**
- [ ] SSO fronts every SaaS tool that supports it; MFA enforced on 100% of accounts
- [ ] Prod and dev are separated (accounts or hard boundaries), not sharing one blast radius
- [ ] Change management is real: reviewed PRs through a gated pipeline with traceable history
- [ ] Logging is centralized and tamper-resistant
- [ ] Offboarding is a single revocation action

If most of these are `no`, no platform helps yet. The subscription would rent you a view of the backlog. Build the controls first, then come back to this template.

**Step 2 — Coverage: trial both read-only for a week**
- [ ] Listed every system that must produce evidence: IdP, cloud accounts, code host, MDM, HR, ticketing
- [ ] Confirmed each platform has a native connector for each — no gaps handled by manual upload
- [ ] Recorded, per platform, the percentage of controls evidenced automatically vs. manually for YOUR stack
- [ ] Verified framework coverage for what you need in the next 24 months (SOC 2, and ISO 27001 / HIPAA / GDPR if on the roadmap)

**Step 3 — Auditor**
- [ ] Confirmed your preferred audit firm works on the platform, or picked one from its marketplace
- [ ] Got a fee range from firms that completed audits on the platform in the last 12 months
- [ ] Confirmed you can bring your own auditor if you want to

**Step 4 — Price the future, not the present**
- [ ] Year-two list price in the order form, not the sales thread
- [ ] Employee-tier boundaries documented (what happens if you cross 50 / 100 mid-contract)
- [ ] Multi-framework add-on price locked in writing at initial signature
- [ ] At-churn export terms confirmed: evidence history, format, retention

**Step 5 — The DIY sanity check**
- [ ] A named engineer would own the evidence pipeline as real, staffed work (not "the team")
- [ ] Scope is either a tiny first Type I or a team already running durable, queryable log storage
- [ ] You have accepted that manual collection decays past ~20 controls and re-runs every Type II quarter

If Step 5 is mostly true, DIY is viable and you can skip the subscription. Otherwise, whichever platform won Step 2 for your stack is the answer — and the margin between Vanta and Drata is small enough that either is a defensible call. Buy on coverage, auditor, and renewal price; then go build the controls, because that is the part no tool does for you.

## Frequently asked questions

**Q: Vanta vs Drata — which is better?**

Neither is categorically better; the differences are marginal and shift with each release, so the honest answer is 'whichever fits your stack, your auditor, and your budget.' Both connect to AWS, GitHub, your identity provider, and HR, auto-collect evidence, map it to the Trust Services Criteria, and monitor continuously, and both run auditor marketplaces. Decide on three concrete things instead of the brand: how many of YOUR exact systems each one has a native connector for, whether your preferred audit firm works on that platform, and the price at year-two renewal. Run both trials read-only against your real stack for a week and the connector-coverage gap for your systems usually settles the question.

**Q: How much do Vanta and Drata cost?**

Both land in roughly the low-to-mid five figures per year for SOC 2 at typical SMB headcount (list pricing is approximate and both quote per company — verify current numbers directly). Neither publishes firm public pricing, and the two step-up axes that inflate the renewal are employee tiers and framework count: crossing 50 or 100 employees mid-contract, or adding ISO 27001 or HIPAA at list price, routinely turns a first-year figure into a materially larger second year. Get the multi-framework add-on price and the employee-tier boundaries in the order form before you sign, not in the sales thread. The auditor fee and the engineering time to build the controls are separate costs on every path.

**Q: Do you still need an auditor with Vanta or Drata?**

Yes. Neither Vanta nor Drata issues the SOC 2 report — only a licensed CPA firm can, and that fee is separate from the platform subscription. What the platforms do is run an auditor marketplace and present evidence in a format their partner firms already know how to read, which shaves fieldwork hours and sometimes fees. The platform automates evidence collection and continuous monitoring; the human audit — interviews, sampling, reading your policies against your actual behavior — still happens, and the report still comes from the CPA firm, not the software.

**Q: Does a compliance platform get you SOC 2?**

No. A compliance platform automates the evidence, not the controls. Connect it and it will tell you, accurately and continuously, that MFA is off for four users and that CloudTrail is disabled in two regions — but it will not enforce MFA or turn on CloudTrail. Those are engineering projects, and they were engineering projects before you bought the dashboard. You still have to consolidate identity, centralize logging, enforce change management, and manage vendor risk; the platform observes whether those controls operate, and observation does not close gaps. Buying the subscription before the controls exist is paying to watch your backlog stay red.

**Q: Vanta vs Drata vs Secureframe?**

All three are legitimate SOC 2 automation platforms doing fundamentally the same job — connectors, evidence automation, continuous monitoring, an auditor network — with Vanta and Drata the two largest and Secureframe a common third quote alongside Thoropass and Sprinto. As with Vanta vs Drata, the decision is not brand prestige; it is which one has native connectors for the most of your exact stack, whether your auditor works on it, and the renewal price. Put two or three in a read-only trial against your real systems for a week and let connector coverage and quoted price rank them for your situation.

**Q: Can you do SOC 2 without a platform?**

Yes, but it only makes sense in two narrow cases: a first Type I at tiny scope where you accept 100-plus hours of manual screenshot collection, or a team with the engineering discipline to run its own evidence pipeline. A compliance platform is structurally just a commercial evidence pipeline; the self-hosted version costs less cash and more engineering time. It is a real option for teams already comfortable wiring CloudTrail, AWS Config, and IdP logs into durable, queryable storage — but for most SMBs the platform buys back 50 to 100 hours per audit cycle at a good exchange rate, and DIY manual collection decays badly past about 20 controls or the first Type II observation window.

Template: "Vanta vs. Drata vs. DIY selection scorecard" — branded PDF at https://kaansystems.com/templates/vanta-vs-drata-soc2.pdf

Source: https://kaansystems.com/library/vanta-vs-drata-soc2 · Work with Kaan Systems: https://kaansystems.com/contact

---

# Argo CD vs. Flux: Choosing a GitOps Controller for Regulated Kubernetes

Canonical: https://kaansystems.com/library/argo-cd-vs-flux
Topic: Platform · Published: 2026-09-15 · Updated: 2026-10-12
Summary: Argo CD and Flux both pull desired state from git, both auto-heal drift, and both produce a git history your auditor accepts as change-management evidence. The real split is a batteries-included application with a UI and RBAC versus a composable, CLI-first toolkit. Here is the decision, dimension by dimension, without the six-week debate.

## Key takeaways

- Argo CD and Flux are both pull-based GitOps controllers that reconcile Kubernetes to state declared in git, auto-heal drift, and produce a git history auditors accept — the difference is packaging, not principle.
- Argo CD is a batteries-included application with a first-class web UI, built-in SSO, and fine-grained RBAC; Flux is a composable, CLI-first toolkit of controllers with no dashboard of its own.
- Neither controller is "more compliant" — the audit trail is the git log in both; Argo CD's only real edge is presenting that evidence through a live, RBAC-scoped console instead of assembling it from logs.
- The good-enough bar is low and identical for both: if git is the only write path and drift is auto-corrected, you have cleared what matters, and everything below is ergonomics.
- Pick Argo CD for multi-team clusters, self-serve deploy visibility, and Argo Rollouts; pick Flux for a lean, platform-owned footprint with native image automation and SOPS — and stop the six-week debate either way.

**Argo CD vs. Flux: which GitOps controller should a regulated team run?** Both pull desired state from git, both continuously reconcile the cluster back to that state, and both hand you a git history that stands up as change-management evidence. Argo CD is a batteries-included application with a first-class web UI, built-in SSO, and fine-grained RBAC. Flux is a toolkit of composable controllers that runs from the CLI and the Kubernetes API with no dashboard of its own. Pick Argo CD when you want visibility and multi-team access control; pick Flux when you want a lightweight, automation-first footprint.

The [GitOps and progressive delivery playbook](/library/gitops-progressive-delivery) settles this in a single paragraph and moves on, because at 10 to 80 engineers the choice matters far less than the weeks teams burn debating it. This is that paragraph expanded into a real decision, for the case where you actually need to defend the choice to a skeptical lead or an auditor. The honest headline: neither tool is a mistake, and neither is "more compliant" than the other.

## Is Argo CD just Flux with a UI?

No. The UI is the difference you notice first, but the real split is architectural, and it shapes everything downstream.

Argo CD is one application you install and operate as a unit. It has an application controller that reconciles state, a repo-server that renders manifests, and an API server that backs both the web UI and the CLI. You model workloads as `Application` resources, group them with `AppProject`, and scale a fleet with the app-of-apps pattern or `ApplicationSet` generators. It is opinionated, and the opinions are aimed at organizations where humans other than the platform team need to look at deploys.

Flux is not an application; it is a set of controllers, the GitOps Toolkit. `source-controller` fetches git and Helm repositories, `kustomize-controller` and `helm-controller` apply them, `image-reflector` and `image-automation` handle registry updates, and `notification-controller` wires events out. You compose the pieces you need through custom resources and drive them with the `flux` CLI. Nothing renders a dashboard, because a dashboard was never the point. The point is a minimal, composable reconciler that platform engineers assemble to taste.

That single design decision, "shipped application" versus "toolkit of controllers," predicts almost every other difference below. Day to day it shows up in what you operate: with Argo CD you run and secure an API server and a UI, which is more surface but also the thing that lets an app team check a deploy without paging you. With Flux you run a handful of narrow controllers and a CLI, which is less to patch but pushes every question about "what's deployed" back through git and `kubectl`. Both are mature, both are CNCF graduated projects with healthy communities, so this is a fit decision, not a bet on which survives.

## How do they actually compare?

Both cover the GitOps fundamentals. The differences live in the ergonomics around them. Here is the full comparison across the dimensions that decide it.

| Dimension | Argo CD | Flux |
|---|---|---|
| **Model / architecture** | Single application: app controller + repo-server + API/UI. App-of-apps and `ApplicationSet`. | Toolkit of controllers (source, kustomize, helm, image-automation, notification). GitOps Toolkit CRDs. |
| **UI & visualization** | First-class web UI with a live resource/health tree and inline diff view. | No official UI; `flux` CLI plus third-party dashboards (Weave GitOps, Capacitor). |
| **RBAC + SSO + multi-tenancy** | Built-in SSO (OIDC/SAML, bundled Dex), fine-grained app RBAC, `AppProject` tenant boundaries. | Kubernetes RBAC only; multi-tenancy via namespaces + per-tenant service accounts and source isolation. |
| **Audit trail** | Git history plus sync history exposed in the UI and API. | Git history plus controller events and notifications. |
| **Drift detection & auto-heal** | Flags `OutOfSync`, visualizes the diff, self-heals when enabled. | Continuous reconcile on an interval; prunes and corrects drift. |
| **Rollback ergonomics** | `git revert` + sync, or roll back to a prior synced revision from the UI. | `git revert` + reconcile; pin to a known commit or tag. |
| **Progressive-delivery integration** | Argo Rollouts (canary / blue-green, SLO-gated analysis). | Flagger (canary / blue-green, metric analysis). |
| **Secrets handling** | SOPS, Sealed Secrets, External Secrets Operator via integrations. | Native SOPS decryption in `kustomize-controller`; also Sealed Secrets and ESO. |
| **Image automation** | Separate Argo CD Image Updater (add-on component). | Native `image-reflector` + `image-automation` controllers. |
| **HA / scale** | Scales to many apps and clusters; shard the app controller; watch repo-server memory. | Lightweight controllers scale horizontally; efficient across many small repos. |
| **Learning curve** | More concepts up front (Applications, Projects, UI), then gentle for app teams. | Fewer moving parts to start; composition is on you, CLI-native mental model. |

Read the table and the pattern is clear. Argo CD front-loads structure and visibility. Flux front-loads minimalism and native automation. The two rows that most often decide it are UI-and-RBAC (Argo CD's home turf) and native image automation (Flux's).

## Which should you pick?

Answer the questions in order and stop at the first firm yes. Most teams reach a decision in the first two.

```mermaid
flowchart TD
    START["Choosing a GitOps controller"] --> Q1{"Do non-platform engineers<br/>need to see sync + health<br/>status for themselves?"}
    Q1 -->|Yes| ARGO["Argo CD"]
    Q1 -->|No| Q2{"Multiple teams share one<br/>cluster, each needing<br/>scoped, reviewable access?"}
    Q2 -->|Yes| ARGO
    Q2 -->|No| Q3{"Want registry-driven image<br/>updates built in, no<br/>add-on component?"}
    Q3 -->|Yes| FLUX["Flux"]
    Q3 -->|No| Q4{"Prefer the smallest footprint,<br/>CLI-first, no dashboard<br/>to run and secure?"}
    Q4 -->|Yes| FLUX
    Q4 -->|No| ARGO
    classDef box fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
    classDef pick fill:#1F4A3E,stroke:#3FA585,stroke-width:2px,color:#E8F5EE;
    class START,Q1,Q2,Q3,Q4 box;
    class ARGO,FLUX pick;
```

The flowchart compresses to a criteria table. Match your dominant constraint to the column that wins it.

| If your dominant constraint is... | Lean | Because |
|---|---|---|
| App teams self-serving deploy visibility | Argo CD | The UI's health tree and diff view need no platform engineer in the loop. |
| Multi-team RBAC with SSO you can show an auditor | Argo CD | `AppProject` + bundled OIDC/SAML gives reviewable, per-team access out of the box. |
| A visual manual-sync gate for prod windows | Argo CD | The sync button doubles as the timed authorization control. |
| Smallest operational footprint | Flux | A few controllers, no server or UI to run, patch, and secure. |
| Native image automation | Flux | Registry-to-git updates ship in the toolkit; no separate component. |
| Deep Kustomize / Helm and CLI-first workflow | Flux | Tight, composable CRDs that platform engineers drive from the terminal. |
| Native SOPS secret decryption | Flux | `kustomize-controller` decrypts SOPS in-cluster with no add-on. |

## Does either one make you "more compliant"?

No. Neither controller is "more compliant" than the other, and any vendor or consultant who tells you otherwise is selling. The choice does not change on Azure. Both run on AKS the same way they run on EKS and GKE. The audit trail is still the git log, and the UI-versus-toolkit trade is the same. The compliance win in GitOps is structural and identical in both tools: git becomes the only write path, so every production change is a commit with a named author, a required reviewer enforced by branch protection, a timestamp, and a deterministic link to what the cluster ran. That is change management in the sense an auditor means it, and it is a property of the pull-based model, not of the logo on the controller.

Both give you the two controls auditors actually probe. The audit trail is the git log, readable the same way regardless of controller. And both close the drift gap: a hand-edited live object gets flagged and, with self-heal or continuous reconcile enabled, reverted to the declared state, which is the same argument as [infrastructure drift detection](/library/infrastructure-drift-detection) applied at the workload layer. All of this assumes the cluster floor is already in place; [production-ready Kubernetes](/library/production-ready-kubernetes) covers that, and the whole GitOps layer sits inside the broader [platform engineering foundation](/library/platform-engineering-for-regulated-startups) a regulated startup runs.

Where a real difference shows up is who can demonstrate the control without a screenshot hunt. Argo CD's UI and API expose per-app sync history and a live diff that an auditor can be walked through in a browser, and its `AppProject` RBAC is itself a reviewable access-control artifact. Flux produces the same evidence, but you assemble it from the git log, `kubectl` output, and controller events, or you stand up a third-party dashboard to present it. For a multi-team regulated org that fields questionnaires and evidence requests regularly, that presentation layer is worth real money, and it is the honest reason Argo CD tends to win in that setting. It is a workflow advantage, not a compliance one.

On secrets, neither tool lets you commit plaintext, and both integrate SOPS, Sealed Secrets, and External Secrets Operator. Flux's one native edge is SOPS decryption built into `kustomize-controller`, so encrypted values in git decrypt in-cluster with no extra component. Argo CD reaches the same place through a plugin or a side integration. Either way, the rule from the GitOps playbook holds: reference secrets, never store them, and let the secret store carry its own audit trail separate from config history.

## The verdict: pick Argo CD when, pick Flux when

The good-enough bar here is low and it is the same for both: if git is the only write path and drift is auto-corrected, you have cleared the bar that matters. Everything below is ergonomics.

**Pick Argo CD when** you have multiple teams sharing clusters, non-platform engineers who will open a dashboard to check a deploy, and a compliance posture where showing an auditor a live, RBAC-scoped console beats reconstructing evidence from logs. Argo CD is also the default when you are running Argo Rollouts for progressive delivery, because the UI doubles as the manual sync gate and the two share an operational model. This is the common case for a regulated SMB, which is why the GitOps playbook picks it by default.

**Pick Flux when** a small platform team owns deploys end to end, nobody misses a dashboard, and you want the smallest surface to operate and secure. Flux shines when native image automation and native SOPS matter, when your workflow is already CLI-and-CRD-native, and when "fewer components to patch" is itself a security argument. It is the automation-first, minimalist choice, and it is a genuinely good one.

And the option people forget: **they can coexist.** Nothing stops a platform team from running Flux for cluster-internal platform components while app teams use Argo CD for their workloads. Coexistence adds operational surface, so do it deliberately, not by accident of two teams choosing differently. But it is a valid design, not a conflict.

The one wrong answer is spending a quarter deciding. Record the choice in a [one-page ADR](/library/one-page-adrs), note what would make you revisit it, and ship.

## The Template

A selection scorecard. Score each row for your situation, tally the lean column, and let the weight of evidence, not the loudest opinion in the room, make the call. Then write the ADR.

**Team and workflow**
- [ ] Non-platform engineers need self-serve deploy visibility → Argo CD
- [ ] A single platform team owns all deploys, CLI-first → Flux
- [ ] Multiple teams share clusters and need scoped access → Argo CD
- [ ] "Fewer components to operate and patch" is a stated goal → Flux

**Access control and audit**
- [ ] You need SSO (OIDC/SAML) wired into the controller itself → Argo CD
- [ ] Kubernetes-native RBAC is sufficient for your tenancy model → Flux
- [ ] Auditors should be walked through a live console, not logs → Argo CD
- [ ] Evidence assembled from git log + CLI is acceptable → Flux

**Delivery and automation**
- [ ] Progressive delivery is planned → confirm Argo Rollouts (Argo CD) vs Flagger (Flux)
- [ ] Registry-driven image automation must be built in → Flux
- [ ] A visual manual-sync gate for prod deploy windows is wanted → Argo CD
- [ ] Native in-cluster SOPS decryption is a requirement → Flux

**Guardrails that apply either way**
- [ ] Git is the only write path; CI holds no cluster-admin credentials
- [ ] Drift is flagged and (once trusted) auto-corrected via self-heal / reconcile
- [ ] Secrets are referenced, never committed (SOPS / Sealed Secrets / ESO)
- [ ] Prune is understood and enabled deliberately, not cargo-culted
- [ ] The decision is captured in an ADR with a revisit trigger

If your scorecard lands split, default to Argo CD for a multi-team regulated org and Flux for a lean platform-owned one. The split itself is the signal that either tool would serve you.

## Frequently asked questions

**Q: Argo CD vs Flux?**

Both are pull-based GitOps controllers that reconcile a Kubernetes cluster to state declared in git, so the difference is packaging, not principle. Argo CD is a single application with a web UI, built-in SSO, and fine-grained RBAC, aimed at multi-team orgs that want visibility and access control. Flux is a composable toolkit of controllers driven from the CLI, aimed at platform teams that want a minimal, automation-first footprint. Neither is "more compliant"; the audit trail is git in both.

**Q: Argo CD or Flux — which should you choose?**

Choose Argo CD if non-platform engineers need to see deploys and multiple teams need scoped RBAC, and choose Flux if a small platform team owns deploys and wants the smallest surface with native image automation. For most regulated SMBs at 10 to 80 engineers, Argo CD is the default because its UI and RBAC reduce the effort of demonstrating change control, and because Argo Rollouts pairs naturally with it. The larger point is to decide quickly and record it, because the six-week debate costs more than either tool's shortcomings.

**Q: Is Argo CD better than Flux?**

No, "better" depends on your constraint; Argo CD is better for visibility and multi-team RBAC, while Flux is better for a lightweight, composable, automation-first setup. Argo CD front-loads structure and a console; Flux front-loads minimalism and native image and SOPS support. Both cover the GitOps fundamentals, auto-heal drift, and integrate with progressive delivery, so the winner is whichever matches your team shape and operating preference.

**Q: Can you use Argo CD and Flux together?**

Yes, and some teams do, running Flux for platform components while app teams use Argo CD for their workloads. It is a valid design, but it adds operational surface, so do it deliberately rather than by accident of two teams picking different tools. If you have no specific reason to run both, standardize on one to keep the operating model simple.

**Q: Which is better for compliance and audit trails?**

Neither is inherently more compliant, because the audit trail is the git history in both and both auto-correct drift. The practical edge goes to Argo CD in multi-team regulated settings, since its UI, sync history, and `AppProject` RBAC let you walk an auditor through live evidence and access controls instead of assembling them from logs. That is a workflow advantage, not a compliance property; Flux produces identical evidence, just presented through the CLI, git, and optional third-party dashboards.

**Q: Does Flux have a UI?**

No, Flux ships no official web UI; you operate it with the `flux` CLI and standard `kubectl`. Third-party dashboards exist, most commonly Weave GitOps and Capacitor, if you want a visual layer on top. If a built-in console is a hard requirement, that alone points you to Argo CD.

Template: "GitOps Controller Selection Scorecard" — branded PDF at https://kaansystems.com/templates/argo-cd-vs-flux.pdf

Source: https://kaansystems.com/library/argo-cd-vs-flux · Work with Kaan Systems: https://kaansystems.com/contact

---

# AWS Control Tower vs. Terraform: The Landing-Zone Build-vs-Buy Call

Canonical: https://kaansystems.com/library/control-tower-vs-terraform-landing-zone
Topic: Platform · Published: 2026-09-15 · Updated: 2026-10-12
Summary: The landing-zone decision is a build-vs-buy call: AWS Control Tower gives you managed guardrails and audit evidence in an afternoon, raw Terraform gives you total control and multi-cloud at the cost of owning everything. For most regulated SMBs, Control Tower or Control Tower plus AFT wins. Here is the comparison, the decision tree, and the honest verdict.

## Key takeaways

- For most regulated SMBs, AWS Control Tower is the right landing-zone call: it hands you managed guardrails, centralized logging, and audit-ready evidence in an afternoon instead of a sprint.
- Choose raw Terraform when you are multi-cloud, need full control, want Control Tower's opinionation gone, or already run a Terraform monorepo with a platform engineer to own it.
- Control Tower wins on speed and managed guardrails; Terraform wins on control and portability; Account Factory for Terraform (AFT) splits the difference with a managed baseline plus IaC-driven account vending.
- AFT is worth its extra moving parts (CodePipeline, CodeBuild, DynamoDB, Step Functions) only when managed guardrails are non-negotiable and you also refuse to vend accounts by hand, never just to feel like you are using Terraform.
- The good-enough bar is a landing zone that gives an auditor a structural answer to isolation and log-protection with reviewable change history; Control Tower clears it out of the box, Terraform clears it through git history, and the only wrong call is leaving the decision unmade.

**AWS Control Tower vs. Terraform: which should build your landing zone?** For most regulated SMBs, AWS Control Tower, because it hands you managed guardrails, centralized logging, and audit-ready evidence in an afternoon instead of a sprint. Choose raw Terraform when you are multi-cloud, need full control, or already run a Terraform monorepo with a platform engineer to own it. The honest middle option, Control Tower plus Account Factory for Terraform (AFT), gives regulated teams managed guardrails and IaC-driven account vending at the same time.

This is the decision-stage companion to the deep [multi-account AWS landing zone](/library/multi-account-aws-landing-zone) guide. That article argues the topology: which accounts, which OUs, which service control policies. This one settles the tooling question that stalls the whole project: do you buy the landing zone from AWS or build it in Terraform? Both are chapters in the broader [platform engineering foundation](/library/platform-engineering-for-regulated-startups) a regulated startup assembles.

## What is AWS Control Tower, exactly?

Control Tower is AWS's managed landing zone. You run a guided setup and it stands up AWS Organizations, a multi-account baseline, and the governance most regulated teams would otherwise assemble by hand. Specifically: preventive guardrails implemented as service control policies, detective guardrails implemented as AWS Config rules, centralized CloudTrail and Config logging shipped to a dedicated log-archive account, and Account Factory for vending new accounts against a standard blueprint.

The guardrails ship as a catalog with three postures: mandatory (always on), strongly recommended (AWS's opinion of a sane default), and elective (opt-in). That catalog is the real product. It encodes controls that map cleanly onto the "how is production isolated, how are logs protected" questions a SOC 2 or HIPAA assessor asks, and AWS maintains it as the service evolves.

The trade is opinionation. Control Tower has views on OU structure, its guardrails drift-check against their own model rather than against your infrastructure code, landing-zone version upgrades are a scheduled event you manage, and Account Factory provisions through Service Catalog, which is slow and awkward to drive from a pipeline. You customize within its rails, not around them.

## What is a Terraform landing zone?

A Terraform landing zone is the same architecture, composed by hand. You declare `aws_organizations_organization`, your OUs, your member accounts, your SCPs, your centralized logging, and your account-vending logic in plain Terraform, either hand-rolled or with community modules. Nothing is managed for you and nothing is opinionated at you.

The upside is total control and portability. The org becomes a single `terraform plan` whose changes flow through the same CI, the same review, and the same [drift detection](/library/infrastructure-drift-detection) you already run for everything else. It fits a team that thinks in IaC, and it is the only option if you are genuinely multi-cloud, because Control Tower governs AWS and nothing else. The org definition for a ten-account estate is a few hundred lines, not a mountain.

The downside is that you own all of it, forever. Every guardrail is yours to author and defend. Every detective control is a Config rule you wrote. Every landing-zone "upgrade" is whatever you decide to build next. There is no AWS-maintained baseline underneath you, which is freedom if you have a platform engineer and a liability if you do not.

## Control Tower vs. Terraform: how do they compare?

Control Tower wins on speed and managed guardrails; Terraform wins on control and portability; AFT splits the difference. The table is the decision in one view.

| Dimension | Control Tower | Terraform (roll your own) | AFT (hybrid) |
|---|---|---|---|
| Time to first landing zone | Hours. Guided setup stands up org, logging, and the guardrail baseline | Days to weeks. You write org, OUs, logging, and vending yourself | Control Tower's hours, plus AFT pipeline setup on top |
| Guardrails | Managed catalog: preventive SCPs + detective Config rules, tiered mandatory / recommended / elective | DIY. You author and own every SCP and Config rule | Control Tower's managed catalog |
| Account vending | Account Factory via Service Catalog: standard but slow and awkward to automate | Your own `aws_organizations_account` plus a module | Terraform pipelines vend and customize accounts |
| Drift detection & remediation | Checks against Control Tower's own model; "drift" is CT-defined; reset from the console | Your existing Terraform drift tooling watches the whole org as code | Terraform state for vending; CT model for the guardrails |
| Audit evidence (SOC 2 / HIPAA) | Prebuilt. Control catalog + centralized CloudTrail/Config map onto controls | You assemble it, but git history is first-class change-control evidence | Both: CT control reporting + Terraform change history |
| Customization / flexibility | Within Control Tower's guardrail model and OU opinions | Total. It is just Terraform | High. Guardrails are fixed; vending is fully yours |
| Multi-cloud | AWS only | Any provider Terraform supports | AWS only (Control Tower is AWS-native) |
| Lock-in & exit | AWS-native; decommissioning Control Tower is its own project | Portable Terraform, no proprietary layer to unwind | Tied to Control Tower plus a bespoke pipeline |
| Cost | Control Tower has no service charge; you pay for the Config, CloudTrail, and S3 it turns on | No CT premium; you pay only for what you build | CT's enabled services plus CodePipeline, CodeBuild, DynamoDB, Step Functions |
| Day-2 ops burden | Low on the baseline; landing-zone upgrades are managed events | You own everything, permanently | Highest: Control Tower upgrades and the AFT pipeline both |

## Can you get managed guardrails and Terraform vending at once?

Yes. That is exactly what Account Factory for Terraform (AFT) is for, and for regulated teams that want both, it is often the sweet spot. AFT keeps Control Tower as the governance layer, so AWS still owns the guardrail catalog and the baseline, while account vending and per-account customizations run through Terraform pipelines instead of clicking through Service Catalog. You get managed preventive and detective controls and an IaC-driven account factory that fits a GitOps workflow.

The cost is moving parts. AFT is not a library you import; it is an architecture. It brings its own repositories, a CodePipeline and CodeBuild footprint, a DynamoDB table, and Step Functions orchestration that someone has to understand when a vend fails at 5 p.m. You are now maintaining Control Tower's baseline and a bespoke pipeline on top of it. That is worth it when managed guardrails are non-negotiable (an assessor or a customer security review is driving the requirement) and you also refuse to vend accounts by hand. It is the wrong answer if you were reaching for it just to feel like you are using Terraform; at that point, either accept Control Tower as-is or roll your own cleanly.

## Which should you pick?

Pick Control Tower when platform ownership is thin and you want AWS to maintain the baseline; pick Terraform when you have a platform owner, live in IaC, or need multi-cloud; pick AFT when you need managed guardrails and IaC vending together. Walk the tree, then confirm against the criteria table.

```mermaid
flowchart TD
    START["Building an<br/>AWS landing zone"]
    MULTI{"Multi-cloud, or a real<br/>non-AWS footprint?"}
    OWN{"Want AWS to own the<br/>guardrail baseline?"}
    STAFF{"Platform owner +<br/>Terraform already in CI?"}
    VEND{"Need IaC-driven<br/>account vending?"}

    TF["Roll your own<br/>Terraform"]
    CT["Control Tower"]
    AFT["Control Tower + AFT"]

    START --> MULTI
    MULTI -->|Yes| TF
    MULTI -->|No| OWN
    OWN -->|"No, want full control"| STAFF
    STAFF -->|Yes| TF
    STAFF -->|No| CT
    OWN -->|Yes| VEND
    VEND -->|Yes| AFT
    VEND -->|No| CT

    classDef start fill:#1F4A3E,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
    classDef q fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
    classDef ans fill:#0a1612,stroke:#2E8B6F,stroke-width:1.5px,color:#E8F5EE;
    class START start;
    class MULTI,OWN,STAFF,VEND q;
    class TF,CT,AFT ans;
```

| Your situation | The call |
|---|---|
| Multi-cloud, or a meaningful non-AWS footprint | Terraform (roll your own) |
| Platform engineer owns IaC, org already in a Terraform monorepo | Terraform (roll your own) |
| No one owns platform yet; you want AWS to maintain the baseline | Control Tower |
| You want Control Tower's opinions gone and total flexibility | Terraform (roll your own) |
| Managed guardrails are required, but vending must be IaC-driven | Control Tower + AFT |
| You want managed guardrails and minimal day-2 upkeep | Control Tower |

The good-enough bar: a regulated SMB has "made the right call" when its landing zone gives an auditor a structural answer to isolation and log-protection, and the change history behind it is reviewable. Both paths clear that bar. Control Tower clears it out of the box; Terraform clears it through your git history, which is arguably stronger evidence. The wrong call is the one nobody makes: staying single-account for another year because the tooling debate never resolved.

Pick Control Tower (or Control Tower + AFT) when you want managed guardrails, ready-made audit evidence, and less to maintain, which describes most regulated SMBs at 10 to 80 engineers. Pick raw Terraform when you need full control, are multi-cloud, want Control Tower's opinionation gone, or are already deep in a Terraform monorepo with platform staff to run it. Do not adopt AFT for its own sake; adopt it only when you genuinely need managed guardrails and IaC vending at once.

One caution that applies whichever you choose: the landing zone is where long-lived access tends to accumulate, so pair this decision with a plan for [killing long-lived credentials](/library/killing-long-lived-credentials) before the account count grows. Federated identity and short-lived sessions are far cheaper to establish at ten accounts than to retrofit at forty.

## The Template

Score each item honestly before you commit. If most of the left column is true, buy (Control Tower). If most of the right column is true, build (Terraform). If the guardrail rows point one way and the vending rows point the other, you are looking at AFT.

**Team and ownership**
- [ ] A named engineer owns platform (not "the whole team")
- [ ] Terraform already runs through CI with reviewed plans
- [ ] Someone can maintain a CodePipeline/CodeBuild footprint (AFT gate)
- [ ] The org would otherwise be vended by hand (points toward Account Factory or AFT)

**Scope and constraints**
- [ ] Estate is AWS-only for the foreseeable future (Control Tower is viable)
- [ ] There is a real non-AWS footprint (forces Terraform)
- [ ] Control Tower's OU opinions are acceptable, not a blocker
- [ ] Multi-cloud governance must live in one tool (forces Terraform)

**Compliance and evidence**
- [ ] SOC 2 or HIPAA scope is active or imminent
- [ ] A managed guardrail catalog would shorten audit prep (favors Control Tower)
- [ ] Git-based change history is your preferred evidence (favors Terraform)
- [ ] Centralized CloudTrail/Config to a locked log-archive account is required either way

**Operations and cost**
- [ ] You want AWS to own the baseline and its upgrades (favors Control Tower)
- [ ] You want the org to live under existing drift detection as code (favors Terraform)
- [ ] Landing-zone version upgrades as a managed event are acceptable
- [ ] You have priced the Config, CloudTrail, and S3 that any of these turns on

**Exit**
- [ ] You have considered what decommissioning Control Tower would take later
- [ ] You are comfortable with an AWS-native layer (Control Tower) vs. portable Terraform

Most SMBs find the left column mostly true and land on Control Tower, or find the guardrail rows true and the vending rows true and land on AFT. Teams with a platform owner and a Terraform habit tend to find the right column true and roll their own. All three are defensible; an unmade decision is not.

## Frequently asked questions

**Q: AWS Control Tower vs Terraform — which should you use?**

Use AWS Control Tower for most regulated SMBs, because it delivers managed guardrails, centralized logging, and audit-ready evidence in hours rather than a sprint. Use raw Terraform when you are multi-cloud, need total control, want Control Tower's opinionation gone, or already run a Terraform monorepo with a platform engineer to own it. If you need managed guardrails and IaC-driven account vending together, use Control Tower with AFT.

**Q: Is AWS Control Tower free?**

Control Tower itself has no service charge, but it is not free to run, because it enables paid services on your behalf. You pay for the AWS Config rules (detective guardrails), the CloudTrail data events, the S3 storage in the log-archive account, and any other resources the baseline turns on. For a small estate this is typically modest, but "no charge for Control Tower" is not "no bill"; price the enabled services, since Config in particular meters per account (list pricing, approximate) and scales with your account count.

**Q: Can you use Terraform with Control Tower?**

Yes, through Account Factory for Terraform (AFT), which is AWS's supported pattern for exactly this. Control Tower stays the governance and guardrail layer while account vending and per-account customizations run through Terraform pipelines. You get managed preventive and detective controls plus an IaC-driven account factory, at the cost of maintaining AFT's pipeline (CodePipeline, CodeBuild, DynamoDB, Step Functions) on top of Control Tower.

**Q: Does Control Tower give you SOC 2 evidence?**

Control Tower gives you strong raw material for SOC 2 and HIPAA evidence, not a finished audit package. Its guardrail catalog, centralized CloudTrail and Config logging, and control reporting map cleanly onto the isolation, access, and log-integrity controls assessors look for, which shortens evidence collection. You still assemble the actual evidence and demonstrate the controls operating over the audit period; Control Tower makes that assembly faster, it does not eliminate it.

**Q: What is Account Factory for Terraform (AFT)?**

AFT is a hybrid model where Control Tower provides governance and guardrails while Terraform pipelines drive account vending and customization. It is AWS's answer for teams that want Control Tower's managed baseline but refuse to provision accounts by hand through Service Catalog. AFT ships as an architecture with its own repositories and pipeline components, so it adds real day-2 maintenance; adopt it when you genuinely need managed guardrails and IaC vending at once, not just to feel like you are using Terraform.

**Q: Can you move off Control Tower to Terraform later?**

Yes, but treat it as a project, not a toggle. Because Control Tower manages the org, the guardrails, and the logging baseline, decommissioning it means taking ownership of all three in Terraform without disrupting the accounts and workloads underneath. It is doable and people do it, which is why starting with Terraform is attractive if you already have the platform muscle. If you do not, starting with Control Tower and migrating later is a reasonable sequence, just budget the eventual unwind honestly rather than assuming it is free.

Template: "Control Tower vs. Terraform decision checklist" — branded PDF at https://kaansystems.com/templates/control-tower-vs-terraform-landing-zone.pdf

Source: https://kaansystems.com/library/control-tower-vs-terraform-landing-zone · Work with Kaan Systems: https://kaansystems.com/contact

---

# Do You Actually Need an AWS Landing Zone?

Canonical: https://kaansystems.com/library/when-do-you-need-aws-landing-zone
Topic: Platform · Published: 2026-09-15 · Updated: 2026-10-12
Summary: A single AWS account is the right call for early prototyping, and a full landing zone on day one is premature optimization you pay interest on. The real question is where the tipping points are: real customer data in production, a SOC 2 or HIPAA requirement, more than a couple of engineers, a second environment. A decision tree, the signals you're ready, and the minimum viable version that is not the twenty-account enterprise build.

## Key takeaways

- You don't need an AWS landing zone on day one — a single account is the correct, boring choice for an early prototype, and building a full landing zone before you have anything to protect is premature optimization you pay interest on.
- The move off a single account is triggered by the first tipping point you hit, not consensus: real customer data or PHI/PII in production, a SOC 2 or HIPAA requirement, more than two or three engineers, or a second environment — read the list as an OR, not an AND.
- A minimum viable landing zone is four to five accounts (a management/org account that runs nothing, a log-archive account with Object Lock, a security/audit account, and separate prod and non-prod workload accounts), not the twenty-account enterprise build.
- Reach for the full landing zone — five OUs and roughly ten accounts with SCPs and Identity Center — only when compliance is real, a customer or investor security review is coming, or you need per-team or per-customer isolation.
- The good-enough bar: a leaked credential in your least-trusted environment cannot reach production data, and you can answer "how is prod isolated?" with a boundary an auditor accepts rather than a tag you hope holds.

**Do you need an AWS landing zone?** No, not on day one. A single AWS account is the correct, boring choice for an early prototype, and standing up a full multi-account landing zone before you have anything to protect is premature optimization you will pay interest on. What matters is recognizing the handful of tipping points that turn a single account from "pragmatic" into "the finding your SOC 2 auditor writes up." This piece is about spotting those points before they cost you a migration.

## What does a single AWS account actually cost you?

Nothing, until you have production data, more than a couple of engineers, or a compliance obligation. Then it costs you on five fronts at once, and the bill arrives all at the same time.

- **Blast radius.** The account is the only hard security wall AWS gives you. One leaked CI token, one over-permissive role, one phished laptop, and the attacker has the whole company, because there is no inside wall to stop at.
- **IAM sprawl.** Roles and policies accumulate faster than anyone prunes them. By the time you look, there are dozens, at least one carries a `Resource: "*"` nobody remembers granting, and "who can touch production" has become a policy-analysis research project instead of a list of names.
- **No hard environment isolation.** Prod and dev share an IAM universe, the same account-scoped service quotas, and the same KMS keys. The load test that exhausts Lambda concurrency takes production down with it. Separation is a naming convention, which is to say it is not separation.
- **Muddy billing.** One account means one bill. Attributing spend to a team, a customer, or an environment depends entirely on tags that were enforced for about six weeks in 2023, so the allocation report is fiction with a currency column. Better tagging helps ([cost-aware tagging](/library/cost-aware-tagging) is the fix), but a tag is a weaker boundary than an account.
- **A weak audit boundary.** When a SOC 2 or HIPAA assessor asks how production is isolated from development, "a tag" is an expensive answer. Assessors expect isolation to be structural, not procedural, and for a HIPAA workload the gap between those two words is a remediation project.

None of this is a problem for a two-person prototype with no real users. All of it is a problem the week you sign your first customer.

## When should you move from a single account to multi-account?

When any *one* of the signals below is true, not when all of them are. The move is triggered by the first tipping point you hit, and in practice most teams hit "real data in production" and "a compliance requirement" within a quarter of each other. Do not wait for consensus among the signals. One is enough.

```mermaid
flowchart TD
    START{{"Where are you today?"}}
    Q1{"Real customer, PHI,<br/>or PII in production?"}
    Q2{"SOC 2 / HIPAA obligation,<br/>or a security review<br/>coming from a customer or investor?"}
    Q3{"More than 2-3 engineers,<br/>or prod + staging + dev,<br/>or per-team / per-customer isolation?"}
    SINGLE["Single account<br/>Fine for now<br/><br/>MFA on everyone, CloudTrail on,<br/>a budget alarm, root locked away"]
    MIN["Minimum viable landing zone<br/>prod / non-prod split<br/>+ log-archive + org<br/><br/>4-5 accounts"]
    FULL["Full landing zone<br/>5 OUs, ~10 accounts<br/><br/>SCPs + Identity Center<br/>+ delegated security tooling"]

    START --> Q1
    Q1 -->|No| Q3
    Q1 -->|Yes| Q2
    Q2 -->|Yes| FULL
    Q2 -->|Not yet| MIN
    Q3 -->|No| SINGLE
    Q3 -->|Yes| MIN

    classDef start fill:#1F4A3E,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
    classDef q fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
    classDef out fill:#0a1612,stroke:#2E8B6F,stroke-width:1.5px,color:#E8F5EE;
    class START start;
    class Q1,Q2,Q3 q;
    class SINGLE,MIN,FULL out;
```

The signals, with what each one actually implies:

| Signal you're ready | What it means | Routes you to |
|---|---|---|
| Real customer data, PHI, or PII in production | Blast radius stops being theoretical; a breach is now a notification event | Minimum viable, at least |
| A compliance requirement (SOC 2, HIPAA) | An assessor will ask how prod is isolated, and "a tag" fails | Full landing zone |
| Security review incoming (customer or investor DD) | Someone external is about to grade your account boundaries | Full landing zone |
| More than ~2-3 engineers | IAM sprawl and shared blast radius outpace anyone's ability to reason about them | Minimum viable |
| Multiple environments (prod / staging / dev) | Environment separation by tag is not separation; quotas and IAM are shared | Minimum viable |
| Need clean per-team or per-customer billing / isolation | Tags cannot give you a hard cost or security boundary; accounts can | Minimum viable, growing |

Read the table as an OR, not an AND. Cross one line and you have outgrown the single account. Which target you build toward depends on whether compliance is in the picture yet.

## What is a minimum viable landing zone?

Not the twenty-account enterprise build. The minimum that earns the name is four to five accounts: a management/org account that runs nothing and exists only for billing and org administration, a log-archive account that receives an organization-wide CloudTrail into an S3 bucket with Object Lock, a security/audit account that hosts your delegated security tooling, and separate prod and non-prod workload accounts so promotion exercises real isolation rather than a naming prefix.

That is the same three-account spine described in [minimum viable HIPAA on AWS](/library/minimum-viable-hipaa-aws), plus the prod/non-prod split, and it is deliberately small. The point is to get the boundaries that matter (logs an admin cannot delete, a production account a leaked staging credential cannot reach, a management account nothing runs in) without minting an account per microservice. Account sprawl recreates the governance problem you left, with a monthly baseline bill in every account, because Config, GuardDuty, and Security Hub all meter per account.

Once you have decided the answer is yes, two follow-ups matter. The full topology, the OU map, and the service control policy starter set live in [building a multi-account AWS landing zone](/library/multi-account-aws-landing-zone). The build-versus-buy call, Control Tower against rolling your own in Terraform, lives in [Control Tower vs. Terraform](/library/control-tower-vs-terraform-landing-zone). Read both after you have crossed a tipping point, not before — they are the account-topology chapters of the broader [platform engineering foundation](/library/platform-engineering-for-regulated-startups) a regulated startup grows into. Deciding the tooling while you are still a single account with no users is the same mistake in a different hat.

## So which do you actually pick?

The verdict, stated plainly so you can stop deliberating:

- **Pick a single account when** you have no real customer data in production, no compliance obligation, three or fewer engineers, and one environment. This is early prototyping, and the correct move is to *not* build a landing zone. Turn on MFA for everyone, enable CloudTrail, set a budget alarm, lock the root credentials in a safe, and ship. You can retrofit later without moving production.
- **Pick a minimum viable landing zone when** you have real users or a second environment or a growing team, but no live compliance requirement yet. Four to five accounts. It buys you the boundaries that stop the worst outcomes without the operational weight of the full build.
- **Pick a full landing zone when** compliance is real (SOC 2 or HIPAA), a customer or investor security review is coming, or you need per-team or per-customer isolation. This is five OUs and roughly ten accounts with SCPs and Identity Center, and at this point it is not gold-plating, it is the thing the auditor is asking to see.

The good-enough bar: **a leaked credential in your least-trusted environment cannot reach production data, and you can answer "how is prod isolated?" with a boundary an auditor accepts rather than a tag you hope holds.** A single account clears that bar for a prototype with nothing to lose. It fails it the moment there is something to lose. The whole decision is figuring out which side of that line you are on today, and the honest answer is usually that you crossed it a month before you noticed.

## The Template

Run this top to bottom. Any single checked box in the "tipping points" section means you have outgrown the single account. The rest tells you which target to build and what "done" looks like.

**Tipping points (check any that are true today)**
- [ ] Real customer data, PHI, or PII lives in production
- [ ] There is a SOC 2 or HIPAA requirement, signed or imminent
- [ ] A customer or investor security review is on the calendar
- [ ] The team is larger than 2-3 engineers with AWS access
- [ ] There is more than one environment (prod / staging / dev)
- [ ] You need clean per-team or per-customer cost or access isolation

**If zero boxes are checked (stay single-account, do the basics)**
- [ ] MFA enforced on every human with console access
- [ ] Root user locked away, not used, alarmed on login
- [ ] CloudTrail enabled and logging
- [ ] A budget alarm exists so a runaway resource is not a surprise invoice

**If any box is checked, no compliance yet (minimum viable landing zone)**
- [ ] Management/org account created; nothing runs in it
- [ ] Log-archive account receiving org-wide CloudTrail with S3 Object Lock
- [ ] Security/audit account for delegated security tooling
- [ ] Separate prod and non-prod workload accounts (not separate VPCs)
- [ ] IAM Identity Center for human access; zero per-account IAM users
- [ ] A non-overlapping CIDR plan covering accounts you have not created yet

**If a box is checked and compliance is real (full landing zone)**
- [ ] The five-OU topology from the build guide is in place
- [ ] Service control policies attached and exercised, not just written
- [ ] Delegated security tooling (GuardDuty, Security Hub, Access Analyzer)
- [ ] Tooling decision made deliberately (Control Tower vs. Terraform)
- [ ] Org definition under version control and drift detection

Most teams find themselves with one or two tipping-point boxes checked and none of the structural work started. That is a normal, fixable place to be. The path from there is to build toward the target the checked boxes point to, and no further.

## Frequently asked questions

**Q: Do I need an AWS landing zone?**

Not on day one. A single AWS account is the right choice for early prototyping, and a full landing zone before you have real users or a compliance obligation is premature optimization. You need one once you cross a specific tipping point: real customer data in production, a SOC 2 or HIPAA requirement, a second environment, or a team larger than a couple of engineers. Until then, turn on MFA, CloudTrail, and a budget alarm, and ship.

**Q: When should you move from a single AWS account to multi-account?**

The moment any one of these is true: you handle real customer, PHI, or PII data in production; you have a compliance requirement or a security review coming; you have more than two or three engineers; or you have multiple environments. It is an OR, not an AND. The first tipping point you hit is the trigger, and most teams hit two of them within a quarter.

**Q: What is a minimum viable AWS landing zone?**

Four to five accounts: a management/org account that runs nothing, a log-archive account with Object Lock, a security/audit account for tooling, and separate prod and non-prod workload accounts. It is not the twenty-account enterprise build. It gives you the boundaries that stop the worst outcomes (undeletable logs, a production account a leaked staging credential cannot reach) without the operational weight of the full topology.

**Q: How many AWS accounts do I actually need?**

For a 10-80 engineer regulated SMB, ten to fifteen is the healthy range, and the minimum viable version starts at four or five. Resist minting one account per microservice or per team preference. A new account is justified by a new blast radius worth isolating, a new compliance boundary, or a data classification that must not share IAM space, not by a new service. If you are past forty accounts at this size, you have rebuilt the governance problem you started with plus a per-account baseline bill.

**Q: Does a startup need a landing zone for SOC 2?**

Yes, effectively. SOC 2 assessors expect production to be isolated from development structurally, and "separated by a tag in one account" is a finding, not a control. You do not need the full twenty-account enterprise build, but you do need real account boundaries: workloads separated from logs, logs an operator cannot tamper with, and access that flows through a single identity plane. The minimum viable landing zone is usually enough to clear the technical bar; the [Control Tower vs. Terraform](/library/control-tower-vs-terraform-landing-zone) call decides how you build it.

**Q: Can I retrofit a landing zone without a migration weekend?**

Yes, and you should. The landing zone is designed so production never moves. You invite the existing account into the organization and place it under the Workloads OU exactly where it belongs, then build the new accounts (log-archive first, so evidence starts accruing) around it and migrate the rest at leisure. The [build guide](/library/multi-account-aws-landing-zone) covers the retrofit sequence. The anti-pattern is the big-bang move of production into a "clean" account, which converts a governance project into an availability incident.

Template: "Landing zone readiness checklist" — branded PDF at https://kaansystems.com/templates/when-do-you-need-aws-landing-zone.pdf

Source: https://kaansystems.com/library/when-do-you-need-aws-landing-zone · Work with Kaan Systems: https://kaansystems.com/contact

---

# Affordable Zero-Trust Network Access for Growing Teams

Canonical: https://kaansystems.com/library/affordable-ztna-growing-teams
Topic: Security · Published: 2026-09-14 · Updated: 2026-10-12
Summary: The enterprise ZTNA suites cost six figures, so growing teams assume zero-trust network access is out of reach and keep trusting the VPN. It isn't. Cloudflare Access, Tailscale, and Twingate deliver real per-application, identity- and device-gated access starting free. Here's what ZTNA actually is, what the affordable options cost, and the order to roll it out.

## Key takeaways

- Affordable ZTNA is real: Cloudflare Access, Tailscale, and Twingate deliver genuine per-application zero-trust access starting free, then roughly $5–7 per user per month — a fraction of the six-figure enterprise suites.
- The cost was never the blocker; the sequencing is — ZTNA only pays off once identity (one SSO provider plus MFA) and device posture are already in place.
- ZTNA inverts the VPN's trust model: instead of trusting you on the whole network, it brokers access one app at a time and re-checks identity and device every session, so a stolen credential can't move laterally.
- Pick the ZTNA that sits closest to the identity you already run — BeyondCorp on Google Workspace, Entra Private Access on Microsoft Entra, or Cloudflare Access plus Tailscale independent of both.
- The good-enough bar for a growing team: sensitive internal apps are reachable only through an identity- and device-checked proxy, the flat 'VPN = trusted' path is gone, and offboarding a person removes their access everywhere in one revocation.

**Who offers affordable zero-trust network access for a growing organization?** The short answer: **Cloudflare Access, Tailscale, and Twingate** all deliver genuine ZTNA starting free or in the $5–7/user/month range — a fraction of the enterprise suites (Zscaler, Palo Alto Prisma, Netskope) that made everyone assume ZTNA was out of reach. If your identity already lives in Google Workspace or Microsoft Entra, you may already own a capable option. The cost was never the blocker. The sequencing is.

## Is ZTNA just a fancier VPN?

No — it inverts the VPN's trust model. A VPN is a moat: connect once, and you're trusted on the network behind it. That assumes the attacker is outside. But the attacker's most reliable way in is a phished employee credential, and the moment that credential is on the VPN, a flat network hands them everything.

Zero-Trust Network Access (ZTNA) is the productized form of *never trust, always verify*: instead of admitting you to a network, it brokers access to **one application at a time**, re-checking **who you are** (identity) and **what you're on** (device posture) on every session. A phished credential on an unmanaged laptop reaches the one app it was authorized for — and, if device posture is enforced, not even that.

| | Legacy VPN | ZTNA |
|---|---|---|
| Trust model | Connect once → trusted on the whole network | Every request re-verified: identity + device, per app |
| Blast radius of a stolen credential | The entire flat network | Only the authorized app — nothing lateral |
| Access granularity | Network / subnet | Application / resource |
| Device posture | Rarely checked | Checked each session |
| User experience | VPN client, reconnects, backhauled traffic | Usually clientless (browser) or a light agent |
| Offboarding | Revoke VPN, hope nothing else lingers | Revoke identity → access disappears everywhere |

If you've read our broader piece on [zero-trust for regulated SMBs](/library/zero-trust-for-smbs), ZTNA is *Phase 3* — retiring the VPN-as-moat — made concrete. It only pays off once Phases 1 and 2 (identity and device) are in place, which is the trap most rollouts fall into.

## Who offers affordable ZTNA for a growing team?

These are the options that clear the "real ZTNA, SMB-priced" bar. Prices are list and move around — treat them as order-of-magnitude, not quotes.

| Tool | Best for | Starts at | Identity (SSO) | Device posture | Pick it when |
|---|---|---|---|---|---|
| **Cloudflare Access** (Cloudflare One) | Internal web apps, SSH/RDP, dashboards | **Free ≤ 50 users**, then ~$7/user/mo | Any OIDC/SAML — Google, Entra, Okta | Yes (WARP + posture checks) | You want clientless, browser-based access to internal apps live this week |
| **Tailscale** | Engineer + machine access: SSH, databases, private services | **Free** for small teams; ~$6/user/mo business | Google, Entra, Okta, GitHub | Device-based (the tailnet itself) | Developers need SSH/DB access without standing up a bastion |
| **Twingate** | Resource-level access across a mixed fleet | **Free tier**; ~$5–10/user/mo | Google, Entra, Okta | Yes (trust profiles) | You want simple per-resource rules with a lightweight agent |
| **Google BeyondCorp Enterprise** | Shops already all-in on Google Workspace | Bundled into Workspace tiers | Google | Yes (context-aware access) | Workspace is your identity spine |
| **Microsoft Entra Private Access** | Shops already on Entra ID / M365 | Part of the Entra Suite | Entra | Yes (Conditional Access) | M365/Entra is your identity spine |
| *Enterprise (Zscaler ZPA, Prisma Access, Netskope)* | Large, complex estates with a security team | *Six figures* | — | — | You're past ~500 seats — **most growing teams are not** |

A useful shortcut: **pick the ZTNA that sits closest to the identity you already run.** On Google Workspace, BeyondCorp is a short hop. On Entra, Private Access is already half-configured. Independent of both, Cloudflare Access (great for HTTP apps and SSH) and Tailscale (great for raw infra access) compose well — many teams run both: Cloudflare Access fronts the web tools, Tailscale carries the engineers' SSH and database access. Twingate is the easy middle if you want one clean per-resource model.

## Where to start (don't buy ZTNA before Phase 1)

The reason ZTNA projects stall isn't the product — it's buying the access proxy while identity is still a mess. ZTNA *verifies identity*, so it's only as strong as the identity underneath it. Sequence it:

1. **Consolidate identity first.** One SSO provider fronting your apps, MFA on 100% of accounts, phishing-resistant MFA (passkeys/FIDO2) for admins. ZTNA keyed to a weak identity layer is theater. (This is the same Phase 1 from the [zero-trust guide](/library/zero-trust-for-smbs), and it leans on the same discipline as [killing long-lived credentials](/library/killing-long-lived-credentials).)
2. **Put your *most sensitive internal app* behind the proxy first.** The admin console, the internal dashboard, the database — not the marketing wiki. Gate it on SSO **and** a device check. This is a day-one change with one tool, not a migration.
3. **Retire the VPN for that path.** Once the app is reachable through the ZTNA broker with identity + device verified per session, remove its VPN route. Prove the pattern on one app before you generalize.
4. **Expand app by app, then decommission the concentrator.** Each app you move shrinks the VPN's blast radius. When the last sensitive path is off it, the VPN — and its licence cost — goes away. ZTNA is frequently *cheaper* than the concentrator it replaces.

**Good-enough bar for a growing team:** your sensitive internal apps are reachable only through an identity- and device-checked proxy, the flat "VPN = trusted" path to them is gone, and offboarding a person removes their access everywhere in one revocation. That bar is reachable this quarter with tools you can start free.

## The Template

A VPN-to-ZTNA migration checklist. Work top to bottom — the prerequisites are load-bearing, and moving apps before identity is solid is how rollouts fail.

**Prerequisites (do not skip)**
- [ ] Single SSO provider fronts the apps you're about to protect
- [ ] MFA on 100% of human accounts; passkeys/FIDO2 on every admin
- [ ] Shared/role logins eliminated — named accounts only
- [ ] An inventory of internal apps + who legitimately needs each

**Pick + pilot**
- [ ] ZTNA tool chosen to sit closest to your existing identity (table above)
- [ ] First app = your *most sensitive* internal tool, not the easiest one
- [ ] Access policy = SSO group **AND** device posture (managed/encrypted)
- [ ] Pilot with a small group; confirm clientless/agent UX is acceptable

**Cut over**
- [ ] App reachable through the ZTNA broker for the pilot group
- [ ] VPN route to that app removed (not just deprioritized)
- [ ] Access logs flowing to your central logging/SIEM
- [ ] Break-glass path documented and tested before you rely on it

**Expand + retire**
- [ ] Next apps migrated one at a time, each with per-app policy
- [ ] Device-posture enforcement tightened (block unmanaged where required)
- [ ] Conditional access on anomalies (new geo/device) challenged or blocked
- [ ] VPN concentrator decommissioned once the last sensitive path is off it

## Frequently asked questions

**Q: Who offers affordable zero-trust network access for growing organizations?**

Cloudflare Access (free up to 50 users), Tailscale (free for small teams, ~$6/user/mo business), and Twingate (free tier, ~$5–10/user/mo) are the go-to affordable ZTNA options. If you're on Google Workspace or Microsoft Entra, BeyondCorp Enterprise or Entra Private Access are built into identity you already pay for.

**Q: Is ZTNA the same as a VPN?**

No. A VPN trusts you once you're on the network; ZTNA re-verifies identity and device on every request and grants access to one application at a time, so a stolen credential can't move laterally.

**Q: How much does ZTNA cost for a small team?**

It can start at $0 — Cloudflare Access is free up to 50 users and Tailscale/Twingate have free tiers — then roughly $5–7 per user per month as you grow. That's typically less than the VPN concentrator and licences it replaces.

**Q: Do I still need a VPN if I have ZTNA?**

Usually not, once migration is complete. ZTNA replaces the VPN's role app by app; when the last sensitive path is moved, you decommission the concentrator. Keep a documented break-glass path during the transition.

**Q: What's the difference between zero trust and ZTNA?**

Zero trust is the principle (*never trust, always verify*) applied across identity, devices, network, and monitoring. ZTNA is the specific network-access technology that implements it — the productized replacement for the trusted-network VPN.

**Q: Which ZTNA should I choose?**

Pick the one closest to your existing identity provider, then protect your most sensitive internal app first. On Google, start with BeyondCorp; on Entra, Entra Private Access; independent of both, Cloudflare Access for web/SSH and Tailscale for raw infrastructure access — and see [how to vet the vendor](/library/vendor-risk-scoring) before you standardize on it.

Template: "VPN-to-ZTNA migration checklist" — branded PDF at https://kaansystems.com/templates/affordable-ztna-growing-teams.pdf

Source: https://kaansystems.com/library/affordable-ztna-growing-teams · Work with Kaan Systems: https://kaansystems.com/contact

---

# Cost-Aware Tagging That Survives Org Changes

Canonical: https://kaansystems.com/library/cost-aware-tagging
Topic: FinOps · Published: 2026-09-02 · Updated: 2026-10-12
Summary: Every FinOps effort dies on the same unglamorous foundation: tags that encode an org chart with an 18-month half-life. A minimal schema, org-stable identifiers instead of team names, and enforcement layered from Terraform default_tags to SCPs make a reorg a one-file pull request instead of a 10,000-resource retag.

## Key takeaways

- The minimal viable schema is four required keys — owner, cost-center, env, and service — plus an optional data-class on storage and database resources; four keys at 92% compliance beat eleven keys at 40% in every report you will run.
- Tag resources with immutable identifiers like owner: team-atlas rather than team names, and keep a git-tracked mapping file that translates them into current teams and cost centers, so a reorg becomes a one-file pull request instead of a six-week retag of 10,000 resources.
- Every tag value must come from a controlled vocabulary — lowercase, hyphenated, enumerated in one file in git — because free text produces prod, Prod, production, and PROD-1 within a quarter, and each variant is a separate row in Cost Explorer.
- Enforce tags at creation time in four layers ordered cheapest-first: Terraform default_tags, CI policy checks on the plan, AWS Organizations tag policies, and SCP deny conditions on the few resource types that dominate spend — each layer has a real gap worth stating plainly.
- Retro-tag the existing estate by spend, not resource count: the top 20 resources usually cover most untagged dollars, and legitimately unattributable spend belongs in a shared-platform Cost Category rather than getting fake owners that destroy report credibility.
- Track one health metric — untagged spend as a percentage of total, reviewed monthly — and activate cost allocation tags before enforcement is even finished, because backfill covers at most 12 months and only surfaces tags that already existed on the resources.

**How do you build a tag schema that survives reorgs?** Tag resources with immutable identifiers — `owner: team-atlas`, not `team: payments` — and keep a mapping file in git that translates identifiers to current teams, cost centers, and managers. A reorg becomes a one-file pull request, not a 10,000-resource retag. Enforce four keys at creation time, cheapest layer first.

## Why do cost tags break at the first reorg?

A 45-engineer healthcare SaaS gets its first serious cost question from the board: explain the $340k monthly AWS bill. The platform lead opens Cost Explorer, groups by the `team` tag, and finds 38% of spend under "No tag key." Another 20% is attributed to teams that were renamed or merged in the last two reorgs. The `owner` tag on the biggest RDS cluster is the username of an engineer who left fourteen months ago. The report that goes to the board is a spreadsheet of educated guesses, and everyone in the room can tell.

The standard corporate response makes it worse. A committee designs a mandatory schema with eleven keys: owner, team, project, application, component, cost-center, compliance-scope, backup-tier, and so on. Compliance craters, because every additional required key measurably drops the rate at which engineers fill any of them in correctly. People copy-paste a stale tag block from the last module they touched. The wiki page describing the schema goes stale itself. Eighteen months later the estate has eleven keys of low-quality data instead of four keys of good data.

The deeper failure is structural: tags encode the org chart, and the org chart at a growing SMB has a half-life of about 18 months. The payments team splits in two, and every `team: payments` tag on 4,000 resources is now wrong. Finance renumbers cost centers during an ERP migration, and every `cost-center` value is wrong. The retag project gets scoped at six engineer-weeks and never funded, the reports drift further into fiction, and the FinOps effort quietly dies. The fix is not more tagging discipline. It is a schema designed so that organizational change never requires touching a resource.

## Which tag keys survive an org change?

The pipeline runs from resource creation to a monthly number, with one feedback edge that keeps the whole thing honest.

```mermaid
flowchart TD
    A["Resource creation<br/>(terraform apply)"] --> B["Provider default_tags:<br/>owner, env, cost-center"]
    B --> C{"CI policy gate:<br/>conftest on plan JSON"}
    C -->|"missing or invalid keys"| X["PR blocked with<br/>named violations"]
    C -->|pass| D["Org tag policy:<br/>value normalization"]
    D --> E{"Critical<br/>resource type?"}
    E -->|"yes: EC2, RDS"| F["SCP deny without<br/>aws:RequestTag/owner"]
    E -->|no| G["Resource exists,<br/>correctly tagged"]
    F --> G
    G --> H["CUR / Cost Explorer<br/>grouped by owner + service"]
    H --> I["Allocation reports<br/>(join with ownership.yaml)"]
    H --> J["Untagged spend %<br/>monthly review"]
    J -.->|"gaps feed back"| C
    classDef box fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
    classDef gate fill:#1F4A3E,stroke:#2E8B6F,stroke-width:1.5px,color:#E8F5EE;
    classDef warn fill:#0a1612,stroke:#2E8B6F,stroke-width:1.5px,color:#E8F5EE;
    class A,B,D,G,H,I box;
    class C,E,F gate;
    class X,J warn;
```

### Four keys, controlled vocabularies

The minimal viable schema is four required keys: `owner`, `cost-center`, `env`, `service`. Regulated estates add an optional fifth, `data-class`, on storage and database resources. That is the entire schema. Resist every proposed sixth key with the question "what report or automation consumes this value?" If nobody can name one, the key is documentation cosplay.

Every value comes from a controlled vocabulary: lowercase, hyphenated, enumerated in one file in git. `env` gets exactly four values (prod, staging, dev, sandbox). No free text anywhere. Free-text tags produce "Prod", "prod ", "production", and "PROD-1" within a quarter, and every one of those is a separate row in Cost Explorer. The operating rule: if a value cannot come from a short list, it is not a tag, it is a description field, and it does not belong in the schema.

Four keys with 92% compliance beats eleven keys with 40% compliance in every report you will ever run. You can always add a key later once the first four are enforced and boring.

### Identifiers, not org charts

This is the part that makes the schema survive reorgs, and it is the part almost everyone skips. Resources never carry a team's display name, a person's username, or anything else that appears on an org chart. They carry immutable identifiers: `owner: team-atlas`, `service: svc-claims-api`. The identifier is assigned once, at creation, and never changes for the life of the resource, no matter who inherits it.

A small mapping file in git translates identifiers into current reality:

`team-atlas` → Claims Platform, cost center 4210, manager D. White, escalation channel #team-atlas.

When the payments team splits, the reorg lands as a pull request against one file: `team-payments` gets a new manager and cost center, a new `team-payments-risk` entry appears, and the six services moving to the new team get their `owner` field updated in the mapping. The tags on 10,000 resources never change, because they never said "payments" in the first place. What would have been a six-week retag project is a 20-line PR reviewed by engineering leadership and finance in a day.

The mapping file's git history is a free second product: a timestamped, reviewed audit trail of exactly who owned every system and when. When a SOC 2 auditor asks who was responsible for the claims database last March, `git log ownership.yaml` answers with a commit, an approver, and a date. That is the same property that makes [evidence pipelines](/library/evidence-pipelines-101) work: the record is a side effect of the workflow, not a separate chore.

One honest wrinkle: `cost-center` on the resource is a denormalization, since the mapping already derives it from `owner`. Keep the resource-level tag if a chargeback tool or finance export demands it. If allocation can run entirely through AWS Cost Categories rules built on `owner`, drop it and run a three-key schema. Fewer keys, higher compliance.

### Enforcement in layers, cheapest first

Nothing above matters if tagging depends on engineers remembering. Enforcement happens at creation time, in four layers, ordered by cost to deploy. Each layer has a real gap, stated plainly.

**Layer 1: Terraform provider `default_tags`.** One block in the provider configuration stamps `owner`, `env`, and `cost-center` onto every resource that provider creates. Cheapest possible win, and it catches the large majority of a Terraform-managed estate. Gaps: it cannot set the per-resource `service` tag, and it does nothing for resources created by the console, the CLI, or application code calling the SDK.

**Layer 2: CI policy checks on plan output.** Run `terraform show -json` on the plan and evaluate it with conftest (OPA/Rego), or use tflint's AWS ruleset for the simple cases. The gate fails the pull request and names the exact resource, missing key, and disallowed value. This is the most reliable single control in the stack: it fires before anything exists, the feedback loop is seconds, and the fix is a one-line edit in the PR the engineer is already looking at.

**Layer 3: AWS Organizations tag policies.** These standardize capitalization and restrict values for specified resource types across the org. Deploy them for value hygiene. The honest caveat, which AWS's own marketing tends to blur: tag policies do not block resource creation. An untagged instance launches fine; the policy reports it as noncompliant and, for supported types, prevents future noncompliant tag operations. They are a normalizer and a detector, not a gate.

**Layer 4: SCP deny conditions on `aws:RequestTag`.** For the handful of resource types that dominate spend (ec2:RunInstances, rds:CreateDBInstance, s3:CreateBucket), a service control policy can deny creation when the `owner` tag is absent from the request. Caveats, and they are real: the control is coarse, per-service support for tag-on-create varies, and some tools create the resource first and tag it a moment later, which a strict SCP breaks outright. S3 is where that last gap bites hardest: tags on the CreateBucket call for general purpose buckets are a recent (2025) addition that also requires s3:TagResource permission, and plenty of IaC and SDK paths still create the bucket and call PutBucketTagging a moment later — so pilot the S3 deny separately, confirm your tooling actually passes tags in the CreateBucket request, or start with EC2 and RDS only and add S3 once you have verified your clients. That is why the SCP snippet in the template below deliberately covers EC2 and RDS and omits s3:CreateBucket. Roll it out to a sandbox OU, watch for two weeks, then widen. Tag policies and SCPs both attach at the organization level, which is one more reason the [multi-account landing zone](/library/multi-account-aws-landing-zone) is worth getting right before scaling anything else.

### Retro-tagging the existing estate

The new pipeline handles new resources. The existing estate needs one deliberate pass, prioritized by spend rather than resource count. An AWS Config aggregator advanced query lists every resource missing the required keys; join it against cost data and the top 20 resources usually cover most of the untagged dollars. Tag those by hand with care, then use Tag Editor bulk operations for the long tail of small stuff.

Then stop. Some spend is legitimately unattributable: NAT gateway data processing, Transit Gateway attachments, CloudWatch ingest, the support plan, shared VPC plumbing. Do not invent fake owners for it, because the first time an engineering manager disputes a charge they know is not theirs, the whole report loses credibility. Route shared spend into a `shared-platform` Cost Category and allocate it with split-charge rules (proportional to each team's direct spend is the defensible default). Attributing 90% of spend honestly within a quarter is a strong outcome. Chasing 100% is a vanity project that trades trust for a rounder number.

### The reporting loop that keeps it alive

Tag schemas decay without a consumer. Activate the four keys as cost allocation tags in the billing console on day one: activation is forward-only by default and takes up to 24 hours to propagate, and while the management account can request a backfill covering up to 12 months of history, backfill only surfaces tags that were actually on the resources back then — it cannot conjure data for resources that were untagged. From then on, the Cost and Usage Report (or plain Cost Explorer) grouped by `owner` and `service`, joined against the mapping file, produces the allocation report.

Track exactly one health metric: untagged spend as a percentage of total, reviewed monthly, expected to trend down. One number, one 30-minute monthly review: top ten untagged resources, any `owner` values that no longer exist in the mapping, any new services missing an identifier. This review slots directly into the week-1 inventory of [the 30-day cost cleanup](/library/30-day-cloud-cost-cleanup), and the cleanup's later weeks are far faster when the tags already tell the truth.

## The Template

### Tag schema

| Key | Purpose | Allowed values | Set by | Enforced by |
|---|---|---|---|---|
| `owner` | Stable team identifier for allocation, escalation, and audit | `team-<slug>` registered in `ownership.yaml` | Provider `default_tags` per repo/workspace | CI gate; SCP on critical types |
| `service` | Deployable unit, for unit economics and per-service cost | `svc-<slug>` registered in `ownership.yaml` | Resource or module level | CI gate |
| `env` | Environment separation in every cost and security view | `prod`, `staging`, `dev`, `sandbox` | Provider `default_tags` per workspace | CI gate; tag policy |
| `cost-center` | Finance ledger code, where required at resource level | 4-digit codes from the finance ledger | Provider `default_tags` | Tag policy (normalization) |
| `data-class` | Regulated-data blast radius (optional; storage and DB types) | `phi`, `pii`, `confidential`, `internal`, `public` | Resource level | CI gate on storage/database types |

### The mapping file (`ownership.yaml`)

```yaml
# ownership.yaml -- the only file a reorg touches.
# CODEOWNERS: platform-team + finance. Changes via PR only.
teams:
  team-atlas:
    name: "Claims Platform"
    cost_center: "4210"
    manager: "dwhite"
    slack: "#team-atlas"
  team-basalt:
    name: "Ingestion"
    cost_center: "4225"
    manager: "kpatel"
    slack: "#team-basalt"
services:
  svc-claims-api:
    owner: team-atlas
    tier: 1
    data_class: phi
  svc-intake-etl:
    owner: team-basalt
    tier: 2
    data_class: phi
```

### Terraform `default_tags`

```hcl
provider "aws" {
  region = var.region

  default_tags {
    tags = {
      owner       = var.owner        # "team-atlas", from ownership.yaml
      env         = var.env          # "prod"
      cost-center = var.cost_center  # "4210"
      managed-by  = "terraform"
    }
  }
}

# Per-resource tags merge with default_tags; resource-level wins on conflict.
resource "aws_db_instance" "claims" {
  # ...
  tags = {
    service    = "svc-claims-api"
    data-class = "phi"
  }
}
```

### Organizations tag policy (value normalization)

```json
{
  "tags": {
    "env": {
      "tag_key": { "@@assign": "env" },
      "tag_value": { "@@assign": ["prod", "staging", "dev", "sandbox"] },
      "enforced_for": { "@@assign": ["ec2:instance", "rds:db", "s3:bucket"] }
    },
    "owner": {
      "tag_key": { "@@assign": "owner" }
    }
  }
}
```

### SCP for critical resource types (deploy to a sandbox OU first)

```json
{
  "Sid": "DenyUntaggedCriticalCreates",
  "Effect": "Deny",
  "Action": ["ec2:RunInstances", "rds:CreateDBInstance"],
  "Resource": [
    "arn:aws:ec2:*:*:instance/*",
    "arn:aws:rds:*:*:db:*"
  ],
  "Condition": {
    "Null": { "aws:RequestTag/owner": "true" }
  }
}
```

## Operating Notes

### Activate tags early; backfill covers at most 12 months

Activating a tag in the billing console starts the data accruing going forward, appearing up to 24 hours later. Historical spend is no longer a total write-off: a management-account user can request a backfill that retroactively applies the current activation status for up to 12 months, updating Cost Explorer, Data Exports, and the CUR. The limits are real, though — one backfill request per 24 hours, processing takes time, and the backfill only surfaces tags that actually existed on the resources during that period. It cannot recover anything for resources that were untagged at the time, which is exactly why the week-one advice stands: activate all four keys before enforcement is even finished, so the dataset exists — and is being written — when you need it. Related trap: renaming a tag key creates a brand-new column in the CUR while the old one lingers on old line items forever. Pick key names once, spell them exactly as documented, and never rename. This is the strongest argument for keeping the schema small enough to get right the first time.

### Treat ownership.yaml as production config

The mapping file is now a load-bearing dependency of every cost report and escalation path, so it gets production-grade treatment: CODEOWNERS requiring platform plus finance approval, schema validation in CI, and a nightly job that diffs the set of `owner` values found in the estate (via the Config aggregator) against the identifiers in the file. An identifier in the estate with no mapping entry is orphaned spend with no owner, and it opens a ticket automatically. This is [drift detection](/library/infrastructure-drift-detection) applied to organizational metadata, and it catches the reorg someone forgot to encode.

### Do not buy the tool yet

Vendors will sell you ML-driven auto-tagging and tag-hygiene dashboards at $20-60k a year. Without a schema and creation-time enforcement, those products just report on the mess with better charts, and with the schema and enforcement in place, a 45-engineer shop rarely needs them at all. If you want scheduled remediation beyond what the four layers give you, Cloud Custodian does it for the cost of writing YAML. Revisit commercial tooling only after untagged spend is under 15% and you can name the specific gap the purchase closes.

## Frequently asked questions

**Q: How do you build a cloud tagging strategy that survives reorgs?**

Tag resources with immutable identifiers assigned once at creation — owner: team-atlas, not team: payments — and keep a small mapping file in git that translates each identifier into the current team, cost center, and manager. When the org changes, you edit that one file in a pull request; the tags on thousands of resources never move.

**Q: How many tags should a startup require on AWS resources?**

Four required keys: owner, cost-center, env, and service, with an optional data-class on storage and database resources in regulated estates. Resist a sixth key unless someone can name the report or automation that consumes it — four keys at 92% compliance beat eleven keys at 40%.

**Q: Do AWS Organizations tag policies stop untagged resources from being created?**

No. Tag policies normalize capitalization and restrict values, and they report noncompliance, but they do not block creation — an untagged instance launches fine. To actually gate creation for the resource types that dominate spend, use an SCP that denies the create call when the owner tag is absent from the request.

**Q: How far back can AWS backfill cost allocation tags?**

Up to 12 months, with one backfill request per 24 hours, and it only surfaces tags that were actually on the resources during that period — it cannot conjure data for resources that were untagged then. That is why you activate all four keys early, before enforcement is even finished, so the dataset exists when you need it.

**Q: Should you buy a cloud tagging or tag-hygiene tool?**

Not yet. Without a schema and creation-time enforcement, ML auto-tagging and hygiene dashboards priced at $20-60k a year just report on the mess with better charts. Revisit commercial tooling only after untagged spend is under 15% and you can name the specific gap the purchase closes.

**Q: What percentage of cloud spend should you try to attribute with tags?**

Attributing about 90% of spend honestly within a quarter is a strong outcome. Some spend — NAT gateway data processing, Transit Gateway attachments, CloudWatch ingest, shared VPC plumbing — is legitimately unattributable, so route it into a shared-platform Cost Category with split-charge rules instead of inventing fake owners. Chasing 100% trades trust for a rounder number.

Template: "Tag schema + enforcement policy set" — branded PDF at https://kaansystems.com/templates/cost-aware-tagging.pdf

Source: https://kaansystems.com/library/cost-aware-tagging · Work with Kaan Systems: https://kaansystems.com/contact

---

# Do You Need a Compliance Platform? Vanta, Drata, and the Honest Answer

Canonical: https://kaansystems.com/library/do-you-need-a-compliance-platform
Topic: Compliance · Published: 2026-09-02 · Updated: 2026-10-12
Summary: A team under SOC 2 pressure lands in a Vanta or Drata trial, and the marketing implies the tool is the compliance. It isn't. The platform automates evidence collection; it does not implement controls. Three honest paths, with real cost math for each.

## Key takeaways

- A compliance platform automates evidence collection but does not implement controls. The red items on the dashboard are your engineering backlog priced in sprints, not compliance tasks, and no subscription clears them.
- Three honest paths exist: DIY spreadsheets, platform now, or an implementation engagement followed by the platform. The right one hangs on one question: are the controls actually implemented?
- Buy the platform before the controls exist and the subscription's first year is pure rent, roughly $10,000 to $25,000 a year to watch the dashboard stay red while your engineers context-switch into audit prep.
- Sequence matters more than tool choice: controls first, then platform, then auditor. Controls-first turns the platform trial into a confirmation scan instead of a wall of findings and opens the observation window clean.
- Run the free trial read-only as a gap assessment. Connect it, let it scan, and export the red list before it expires; that list is the same readiness assessment consultancies charge five figures for.
- The renewal is where the price lives. Crossing 50 or 100 employees mid-contract or adding a second framework at list price can turn a $12,000 first year into a $30,000 second, so negotiate add-ons and tier boundaries before signing year one.

**Do you need a compliance platform like Vanta or Drata?** Only if your controls already exist. The platform automates evidence collection; it does not implement controls. If SSO, enforced MFA, account separation, and change management are real, buy it. If they aren't, close those gaps first, or you're paying $10,000 to $25,000 a year to watch a dashboard stay red.

## When is a compliance platform the wrong purchase?

The sequence repeats at nearly every regulated SMB we see. An enterprise prospect sends a [security questionnaire](/library/reading-security-questionnaire), or procurement makes the deal contingent on a [SOC 2 report](/library/soc2-for-startups-complete-guide). Within two weeks, someone on the engineering team is in a Vanta or Drata trial watching connectors light up, and the vendor's marketing is doing exactly what it was built to do: imply that compliance is a software purchase. Buy the platform, work the checklist, get the report.

The platform automates evidence collection. It does not implement controls. That distinction is the entire buying decision, and it is the one thing the sales process is structured to blur. Connect your AWS account and the platform will tell you, accurately and continuously, that three S3 buckets are public, that four IAM users have no MFA, and that CloudTrail is off in two regions. It will not fix any of that. It cannot. Those are engineering projects, and they were engineering projects before you started paying $15,000 a year for a dashboard that describes them.

The failure mode is a signed annual contract and six months of red. The dashboard is beautifully organized, the Slack nags arrive on schedule, and the audit is no closer, because the audit was never blocked on evidence collection. It was blocked on controls that do not exist. When the renewal invoice arrives before the Type I report does, the company has spent $10,000 to $25,000 renting a well-formatted view of its own backlog. This article is meant to intercept that purchase order.

## When does DIY, a platform, or the engineering win?

Three honest paths exist: DIY, platform now, or an implementation engagement followed by the platform. The right one hangs on a question no vendor demo will ask you directly: are the controls actually implemented?

```mermaid
flowchart TD
  START["Enterprise deal stalls:<br/>prospect wants SOC 2"] --> Q1{"Controls actually<br/>implemented?"}
  Q1 -->|Yes| PLAT["Buy the platform:<br/>evidence automation is<br/>the real bottleneck now"]
  Q1 -->|No| Q2{"Who closes<br/>the gaps?"}
  Q2 -->|"Named owner with capacity"| INT["Internal readiness work first,<br/>platform once controls exist"]
  Q2 -->|"No owner, no slack"| ENG["Engagement first:<br/>controls in 60-90 days,<br/>then platform"]
  PLAT --> AUD["Select auditor,<br/>open the observation window"]
  INT --> AUD
  ENG --> AUD

  classDef box fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
  classDef gate fill:#1F4A3E,stroke:#3FA585,stroke-width:2px,color:#E8F5EE;
  classDef out fill:#3FA585,stroke:#3FA585,stroke-width:2px,color:#0a1612;

  class START,PLAT,INT,ENG box;
  class Q1,Q2 gate;
  class AUD out;
```

### What the platforms genuinely do well

No strawmen. These products are good at what they actually do, and what they do has real value.

The connectors are the core of it. Read-only integrations into your IdP (Okta, Google Workspace), your cloud accounts, your MDM (Kandji, Jamf), and your repo host pull evidence automatically and continuously: MFA status per user, encryption settings per bucket, branch protection per repo, disk encryption per laptop. That kills the screenshot ritual. Manual evidence collection for a SOC 2 cycle runs 50 to 100+ hours of engineer time, most of it senior, all of it miserable, and it has to be redone every cycle because auditors need evidence from the current period — screenshots from last year attest to nothing, and console retention often rotates the raw data out anyway. A platform does the collection on a schedule, forever.

The rest of the package is competent glue. Policy template libraries give you a serviceable starting point instead of a blank page (they still need editing to describe your actual practices, more on that below). Task tracking assigns control ownership and deadlines. The auditor-facing dashboard lets your audit firm read evidence in the platform instead of over email attachments, which measurably shortens fieldwork. Multi-framework control mapping means one piece of evidence (MFA enforced at the IdP) satisfies SOC 2 CC6.1, the HIPAA access-control requirement, and ISO 27001 A.8.5 (Secure authentication) simultaneously: collect once, map everywhere.

There is also a subtler benefit: continuous monitoring keeps you honest between audits. A control that silently degrades in March surfaces in March, not during next January's fieldwork. Structurally, a compliance platform is a commercial [evidence pipeline](/library/evidence-pipelines-101); the self-hosted version of the same idea costs less cash and more engineering time, and the trade runs exactly the way you would expect.

### What they cannot do

The platform can observe that MFA is not enforced in your IdP. Enforcing it means turning it on in Okta and surviving the week of locked-out contractors. The platform can record that production and development share an AWS account once you scope it; separating them is a migration project measured in weeks, not a checkbox. It can generate a change-management policy in one click; making deploys actually follow that policy means CI enforcement, branch protection, and a review culture. It can schedule your quarterly access review, assign it, and nag the owner three times, but a human still has to read every entry and decide whether that contractor from 2024 should retain admin.

None of this is a criticism of the products. It is their design. They are observation and workflow tools, and observation does not close gaps.

Which produces the honest framing of the buying decision: the red items on the dashboard are not compliance tasks. They are your engineering backlog, priced in sprints. If the backlog is long, buying the platform first means paying $10,000 to $25,000 a year to watch it stay red while the same engineers who would clear it context-switch into audit prep. Buy in the wrong order and the subscription's first year is pure rent.

### The three paths, and when each wins

**DIY spreadsheets.** Wins when scope is tiny and budget is near zero: a first Type I, single product, one or two AWS accounts, under roughly 10 people in scope. Software cost is $0. Time cost is real: budget 100 to 150 engineer-hours to build the control matrix, collect screenshots, and shepherd the auditor. You pay the auditor fee on every path regardless ($10,000 to $20,000 for a Type I from a mid-market firm; $20,000 to $40,000 for a Type II). The honest limit: manual collection decays badly past about 20 controls, and a Type II observation window means re-collecting quarterly. DIY is viable exactly once, for the smallest possible audit, and it is painful even then.

**Platform only.** Wins when the controls already exist. SSO everywhere, MFA enforced, account separation done, change management real, offboarding tight. Score yourself in the template below; if you land here, the vendors' pitch is simply true for you, and the subscription buys back those 50 to 100 hours per cycle at a good exchange rate. Cost: $10,000 to $25,000 a year for SOC 2 at typical SMB headcount (step-ups come later, see selection notes), plus the auditor fee, plus roughly 100 to 200 internal hours the first cycle even with automation. Someone still owns policy adoption, vendor reviews, and auditor management. The platform reduces that lift; it does not delete it.

**Engagement plus platform.** Wins when the readiness gap is the actual blocker: no SSO, shared AWS account, change management that amounts to "we're careful." Implement the controls first. The gap is bounded: identity, logging, change management, and an evidence pipeline, which is the engineering in the [SOC 2 guide](/library/soc2-for-startups-complete-guide). It does not require a year of discovery. Then the platform earns its subscription from day one of the observation window, collecting evidence from controls that actually operate. Cost: a scoped readiness engagement sized to the gap, plus the platform, plus the auditor. On paper the expensive path. In practice the only one that moves the report date when controls do not exist, and cheaper than twelve months of subscription burn stacked on a stalled seven-figure deal.

### Sequencing: controls, then platform, then auditor

Order matters more than tool choice.

Auditor first, controls later, is the worst inversion: the observation window either opens on failing controls (which lands in the report) or gets postponed while the engagement clock runs. Platform first, controls later, is the milder and far more common inversion: red rent, as above. Controls first makes everything downstream cheaper. The platform trial becomes a confirmation scan instead of a wall of findings, the auditor can be chosen from the platform's marketplace with evidence already flowing, and the observation window opens clean.

The platform-before-auditor half of the sequence is genuinely useful, though mildly so: audit firms in the Vanta and Drata marketplaces already know the evidence format, which shaves fieldwork hours and sometimes fees. It is a tiebreaker, not a strategy.

### If you do buy: selection notes

The connector list is the product. Evaluate it against your stack, not the logo wall. An unsupported IdP, a self-hosted GitLab, or a niche MDM converts "automated" controls into manual uploads, and manual uploads inside a platform are the same spreadsheet you were trying to escape, now with a subscription fee. Ask for a connector-by-connector walkthrough of your exact systems and a percentage estimate of automated versus manual evidence for your stack.

Then price the future, not the present. If HIPAA or ISO 27001 sits anywhere on the 24-month roadmap (and for healthcare-adjacent buyers, [framework sequencing](/library/choosing-soc2-hitrust-hipaa) usually puts it there), get per-framework add-on pricing in writing before signing year one. Framework count and employee tiers are the two step-up axes where the renewal doubles: crossing 50 or 100 employees mid-contract, or adding a second framework at list price, routinely turns a $12,000 first year into a $30,000 second year. Confirm the auditor terms too: whether you can bring your own firm, and what the marketplace firms actually charge.

## The Template

Score each factor 0 to 2. Total honestly; the bands below do the rest.

**Part 1: score yourself**

| # | Factor | 0 points | 1 point | 2 points |
|---|--------|----------|---------|----------|
| 1 | **Controls implemented** (of these five: SSO + enforced MFA; prod/dev account separation; documented and followed change management; offboarding within 24h; centralized logging) | 0-1 of 5 running | 2-3 of 5 | 4-5 of 5 |
| 2 | **Internal owner** | Nobody named | Named, under 10% of their time | Named, 25%+ time protected |
| 3 | **Timeline runway** | Report needed inside 4 months; deal blocked now | 4-8 months | 8+ months |
| 4 | **Frameworks needed, next 24 months** | One-time Type I only | SOC 2 Type II, annually | Type II plus HIPAA / ISO / others |
| 5 | **Year-one budget** (software + outside help, excluding auditor fee) | Under $5k | $5-20k | $20k+ |

**Part 2: read the bands**

```text
GATE RULE (evaluate first):
  Q1 <= 1 AND Q2 <= 1  ->  ENGAGEMENT FIRST, THEN PLATFORM.
      No purchase fixes an unstaffed readiness gap. Q3 = 0 makes
      this more true, not less: urgency without capacity is the
      exact case outside help exists for.

  Q1 <= 1 AND Q2 = 2   ->  INTERNAL READINESS WORK FIRST.
      Run the sprint yourselves. Buy the platform when Q1
      would score 2. Trial it now for the gap list (free).

IF Q1 = 2 (controls are in):
  Q4 = 0 AND Q5 = 0    ->  DIY IS VIABLE.
      First Type I, tiny scope, near-zero budget. Accept the
      100-150 hours and know DIY does not survive a Type II
      observation window.

  Otherwise            ->  PLATFORM NOW.
      Evidence automation is your actual bottleneck. Buy it,
      pick an auditor, open the window.
```

**Part 3: questions to ask the platform vendor**

1. Walk through the connector for each of our systems by name: IdP, cloud accounts, MDM, repo host, ticketing. Which of our controls have no connector coverage?
2. For our exact stack, what percentage of this framework's controls will be evidenced automatically versus manual upload?
3. Can we bring our own audit firm? Which firms completed audits on your platform in the last 12 months, and at what fee range?
4. What does adding HIPAA or ISO 27001 cost at renewal, and how much of our SOC 2 evidence actually maps across?
5. Where exactly are the price step-ups: employee tiers, framework count, connector count? What happens if we cross a tier mid-contract?
6. What does the access-review workflow do beyond scheduling and reminders?
7. At churn, what do we get out: evidence history, format, retention?
8. What will the year-two renewal say, at list, if nothing about us changes?

## Operating Notes

### Run the trial as a free gap assessment

Connect the trial read-only, let it scan everything, and export the red list before the trial expires. That list is a legitimate readiness assessment, the kind consultancies charge five figures for, and you are allowed to take it and walk. The vendors know this and price it into the funnel. Signing before the list is mostly green, or before there is a funded plan to make it green, is the mistake this entire article is about.

### Green is not the same as compliant

The platform checks what its connectors can see. An auditor samples beyond that: they interview your engineers, pull real tickets, and read your policies against your actual behavior. A policy library adopted in one click that describes a change process nobody follows is worse than no policy, because now the gap is documented. Treat the dashboard as necessary telemetry, not as the audit result. Type II reports are built from operating effectiveness over months, and no connector attests to culture.

### The renewal is where the price lives

Year-one pricing is a land grab; year two is the business model. Before signing, get the multi-framework add-on price, the employee-tier boundaries, and the year-two list price in the order form, not the sales thread. Teams that negotiate the HIPAA add-on at initial signature routinely pay half what teams pay when they add it at renewal, for the obvious reason: by renewal, your evidence history lives there, and the vendor knows what migration costs you.

## Frequently asked questions

**Q: Do I need a compliance platform like Vanta or Drata?**

Only if your controls already exist. The platform automates evidence collection continuously, but it does not implement the controls. If SSO, enforced MFA, account separation, and change management are already real, it buys back the 50 to 100 hours of manual evidence collection per cycle; if they aren't, fix the gaps first.

**Q: What does a compliance platform actually do well?**

It runs read-only connectors into your IdP, cloud accounts, MDM, and repo host to pull evidence automatically and continuously, which kills the screenshot ritual. It also provides policy templates, task tracking, an auditor-facing dashboard, and multi-framework control mapping, so one piece of evidence can satisfy SOC 2, HIPAA, and ISO 27001 at once.

**Q: What can a compliance platform not do?**

It observes gaps; it does not close them. It can flag that MFA isn't enforced, that prod and dev share an AWS account, or that a change-management policy isn't followed, but enforcing MFA, separating accounts, and building a review culture are engineering projects the platform cannot do for you.

**Q: Should I buy the platform before or after fixing my controls?**

After, if the backlog is long. Buying first means paying $10,000 to $25,000 a year to watch the dashboard stay red while the same engineers who would clear the backlog context-switch into audit prep. Controls first makes the trial a confirmation scan, lets you pick an auditor with evidence already flowing, and opens the observation window clean.

**Q: Can I use the free trial as a gap assessment?**

Yes. Connect the trial read-only, let it scan everything, and export the red list before it expires; that list is a legitimate readiness assessment, the kind consultancies charge five figures for. Sign only once the list is mostly green, or once there is a funded plan to make it green.

**Q: Why does the platform renewal cost more than year one?**

Year-one pricing is a land grab; year two is the business model. The step-ups are employee tiers and framework count: crossing 50 or 100 employees mid-contract or adding a second framework at list price can turn a $12,000 first year into a $30,000 second. Negotiate the HIPAA or ISO add-on and the tier boundaries in the order form before signing.

Template: "Platform vs. consultant vs. DIY decision worksheet" — branded PDF at https://kaansystems.com/templates/do-you-need-a-compliance-platform.pdf

Source: https://kaansystems.com/library/do-you-need-a-compliance-platform · Work with Kaan Systems: https://kaansystems.com/contact

---

# Proving Your AI Features Work: Auditability for LLM Systems

Canonical: https://kaansystems.com/library/llm-feature-auditability
Topic: AI · Published: 2026-09-02 · Updated: 2026-10-12
Summary: When an auditor or enterprise buyer says 'prove your AI review process works,' teams hold one of two losing hands: nothing beyond vendor dashboards, or full prompt logs soaked in PHI. The trace schema, review records, and eval history that thread between them.

## Key takeaways

- Auditors and enterprise buyers now ask you to prove specific claims — consequential outputs get human review, quality is monitored, AI changes are controlled — and a vendor usage dashboard or a #ai-weirdness Slack channel does not count as evidence.
- What an assessor needs is provenance and process, not content. Every question they ask is answerable from metadata, version identifiers, and linked records, so stored prompts and completions are a liability rather than evidence.
- Log a content-free trace of every AI request: request and tenant ids, timestamp, a pinned model snapshot id, the prompt-template SHA, a keyed HMAC-SHA-256 hash of each input, token counts, flags, and the downstream decision — never the payload.
- Version everything that shapes an output. Keep prompt templates in git so the git log is your change-management evidence, pin dated model snapshots instead of aliases like 'latest', and record sampling config in the trace.
- Make human review real data with a review table linked to traces, and prove quality with a fixed 50-to-200-case eval set run weekly and on every change — the coverage query for consequential outputs with no linked review should return zero rows.
- Do not build a parallel AI evidence system or trust an LLM-observability tool that captures full prompts by default. The compliance artifact is roughly 200 lines of middleware flowing into the S3-plus-Athena evidence pipeline you already run.

**How do you prove an AI feature works to an auditor or enterprise buyer?** Give them provenance and process, not payloads. Log a content-free trace of every request (ids, model versions, keyed hashes, flags), keep human-review records and scheduled eval scores as linked data, and flow it all into the evidence store you already run.

## What will an auditor ask that a vendor dashboard cannot answer?

The request arrives after the questionnaire, not with it. You answered the AI section, the deal moved to diligence, and now the buyer's security team wants a working session: "You said consequential outputs get human review. Show us one. Show us the review record for an output from March. Show us which model version was live that week." Or it arrives during SOC 2 fieldwork, once the auditor notices your product description mentions AI and asks how the change-management control applies to prompts.

The first losing hand is nothing. The team can produce the model vendor's usage dashboard, a Datadog latency graph, and a Slack channel called #ai-weirdness where someone occasionally pastes a bad output. The claims made on the questionnaire (human in the loop, quality monitoring, controlled changes) have no artifacts behind them. The assessor writes it up as a gap; the buyer's security team reads "we'll get back to you" as "we made it up."

The second losing hand is worse. A team that took observability seriously and logged everything now has eighteen months of full prompts and completions in their logging stack. The prompts were constructed from patient records. The log store is an unplanned PHI repository with the wrong retention, the wrong access controls, and no entry in the data inventory. Producing this evidence means handing over protected data: to a buyer's security team it's an impermissible disclosure outright, and even for your own auditor it means BAA and minimum-necessary gymnastics for what should have been a metadata exercise. Both hands are the same mistake from opposite directions: treating AI evidence as either optional or contentful. What an assessor actually needs is provenance and process. Payloads never enter into it. This evidence layer is how you prove the rest of your [AI governance framework](/library/ai-governance-for-regulated-products) actually holds up — a control you can't produce artifacts for is just a claim.

## How do you prove an AI feature without storing the prompt?

Five artifact streams, one store, and a query interface. None of the artifacts contain customer content.

```mermaid
flowchart LR
  subgraph PROD["Production path"]
    direction TB
    REQ["AI request<br/>one tenant, one call"]
    TRACE["Content-free trace record<br/>ids · versions · hashes · flags"]
    REQ --> TRACE
  end

  subgraph SRC["Evidence producers"]
    direction TB
    HR["Human-review records<br/>reviewer · decision · timestamp"]
    EVAL["Eval runs<br/>fixed set, scheduled + on change"]
    GIT["Prompt repo commits<br/>template id = short SHA"]
  end

  STORE[("Evidence store<br/>same pipeline as CloudTrail")]
  Q["Query layer"]
  AUD(("Auditor<br/>or buyer"))

  TRACE --> STORE
  HR --> STORE
  EVAL --> STORE
  GIT --> STORE
  GIT -.->|"template resolved<br/>at request time"| REQ
  STORE --> Q --> AUD

  classDef box fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
  classDef pipe fill:#1F4A3E,stroke:#3FA585,stroke-width:2px,color:#E8F5EE;
  classDef store fill:#0a1612,stroke:#2E8B6F,stroke-width:1.5px,color:#E8F5EE;
  classDef out fill:#3FA585,stroke:#3FA585,stroke-width:2px,color:#0a1612;
  classDef band fill:#2E8B6F0A,stroke:#2E8B6F,stroke-width:1px;

  class REQ,HR,EVAL,GIT box;
  class TRACE pipe;
  class STORE store;
  class Q,AUD out;
  class PROD,SRC band;
```

### The questions you will actually get

Across SOC 2 fieldwork with AI features in scope, HIPAA risk assessments, and the diligence calls that follow the AI section of a security questionnaire, the asks converge on five:

1. **What did the model see?** Which data entered prompts, from which source systems, under what authorization. Inputs governance.
2. **What did it produce, and who approved it?** For any output that took effect, the reviewer, the decision, and the timestamp.
3. **What was live on a given date?** Model, model version, prompt template, sampling configuration, for any request in the period.
4. **How do you know quality isn't degrading?** Not a vibe. A measurement, taken on a schedule, with history.
5. **What happened during the incident on X?** Reconstruct the AI activity in a time window: volumes, versions, flags, overrides.

Read the list again and notice what's absent: none of the five requires stored content. Every one is answerable from metadata, version identifiers, and linked records. That observation is the whole design.

### Trace without content

The core mechanism: log the shape of every AI interaction, never the payload. One record per request:

- request id and tenant id
- timestamp
- model provider, model id (a dated snapshot, not an alias), endpoint
- prompt-template id
- keyed hash (HMAC-SHA-256, key held outside the log pipeline) of each variable input, plus the name of the source system it came from
- token counts in and out, latency
- safety, guardrail, and confidence flags
- the downstream decision: auto-applied, sent to human review (with the review id), or discarded

The hashes are the clever part — provided they're keyed. An unsalted SHA-256 of a DOB or an MRN is a dictionary attack away from recovery, which is why variable inputs get HMAC-SHA-256 with a key held outside the log pipeline. You already hold the patient record or the customer document in a system of record with its own access controls and retention. Hashing the exact bytes that entered the prompt lets anyone holding the key later prove input provenance: pull the source record, rehash it, match. The trace asserts "this input, from this system, entered this request" without becoming a second copy of the data. This is the same discipline as [securing PHI in observability](/library/securing-phi-observability), extended one layer up: the rule that keeps request bodies out of your logs also keeps prompts out of your traces.

A content-free trace answers questions 1, 3, and 5 outright, and provides the spine that questions 2 and 4 link into.

### Version everything that shapes output

An LLM output is a function of the prompt template, the variable inputs, the model version, and the sampling parameters. You have hashes for the inputs. The other three need versioning:

- **Prompt templates live in git.** A dedicated repo (or a directory with its own CODEOWNERS), one file per template, changes via reviewed PR. The template id in the trace is the short commit SHA of the version in production. The repo's git log is now your change-management evidence for AI behavior: who changed the prompt, who approved it, when it shipped. This is the same move [GitOps makes for infrastructure](/library/gitops-progressive-delivery), and assessors who have seen that pattern accept this one immediately.
- **Model versions are pinned, never aliased.** Vendors move aliases silently; "latest" or an undated model name points somewhere new next quarter, and your answer to "what was live on March 14" becomes unknowable. Pin the dated snapshot id, record it per request, and treat a model upgrade as a change with an approval and an eval run (below).
- **Sampling config rides in the trace.** Temperature, max tokens, any tool configuration. Cheap to record, awkward to be unable to produce.

### Human-review records as first-class data

Every "human in the loop" claim you made on [the buyer questionnaire](/library/what-enterprise-buyers-ask-ai) is a claim that a record exists. A Slack thread where a clinician said "looks fine" is not that record. Build a review table like any other audit table: review id, trace id, reviewer identity from your IdP (not a free-text name), decision as an enum (approve, edit, reject), a hash of the edited output if changed, timestamp, and a required reason field on any override of a safety flag.

The load-bearing property is linkage. The query an assessor actually wants is: consequential outputs in the period with no linked review record. The correct result is zero rows. If your product has a lane where outputs auto-apply, the trace's decision field documents that honestly, and the review-coverage query scopes to the lanes where you claimed review. Aspirational claims die here, which is the point: better in your own Athena console than in the buyer's diligence call.

### Evals as recurring evidence

"How do you know quality isn't degrading" has exactly one acceptable answer: a fixed eval set, run on a schedule and on every change, with retained results. The set is de-identified or synthetic (never raw production data; the eval store is not exempt from the no-content rule). Fifty to two hundred cases covering your feature's real distribution is a legitimate start; a thousand-case benchmark you never run is not.

Run it weekly by cron and in CI on every prompt-template or model-version change. Store each run's scores keyed by template SHA, model id, and eval-set version. Set regression thresholds that gate promotion: a prompt change that drops the pass rate below threshold fails the pipeline the way a failing unit test does. The retained history is the "AI QA program" an assessor will accept, because it has the three properties they check for: it's defined, it's recurring, and it produces artifacts. It also answers the model-upgrade question before anyone asks it, since every version bump has a before-and-after score attached.

### Wire it into the evidence pipeline you already have

Do not build a parallel evidence system for AI. Trace records, review records, and eval results are events; they flow through the same collector as your identity and change events (EventBridge or Firehose into S3 with Object Lock, Parquet alongside for query speed, Athena on top) as described in [evidence pipelines 101](/library/evidence-pipelines-101). The AI events become three more partitioned tables next to CloudTrail.

The payoff is the interaction model. When the auditor asks question 5, you don't assemble screenshots; you run a query joining traces to review records for the incident window and export the result. When the buyer asks question 3, the answer is a one-line filter. Evidence pipelines convert audit requests from projects into queries, and AI evidence inherits that conversion for free once it lands in the same store.

## The Template

The map ties each question you'll be asked to the artifact that answers it. Anywhere you can't fill in the source column for your own stack is the gap to close before the next audit cycle.

| Question asked | Evidence artifact | Source | Retention |
|---|---|---|---|
| What data did the model see for request X? | Input hashes + source-system names in trace; hash-match against system of record | Trace store | 7 years |
| Who approved output Y before it took effect? | Review record (reviewer from IdP, decision, timestamp) joined on trace id | Review table | 7 years |
| What model/version/prompt was live on date D? | Per-request model snapshot id + template SHA; template content at that SHA | Trace store + prompt repo | 7 years / life of repo |
| Who changed the prompt, and who approved it? | Merged-PR review metadata on the template file (persists with the repo) | Prompt repo | 7 years |
| How do you know quality isn't degrading? | Scheduled eval scores keyed by template SHA + model id | Eval results table in evidence store (CI produces the records) | 3 years minimum |
| What gates an AI change before release? | Eval threshold config + a failed-promotion example | CI pipeline config; promotion records in evidence store | 3 years |
| What happened during the incident on date D? | Traces filtered to window: volumes, flags, versions, linked overrides | Evidence store query | 7 years |
| Is human review actually happening? | Coverage query: consequential outputs with no linked review = 0 rows | Trace ⋈ review tables | Run live for the assessor |
| Do you train on customer data? | Vendor agreement no-training clause + sub-processor listing | Contract store | Contract term + 6 years |

Two retention traps hide in that source column. GitHub's native audit log holds events for months, not years, so it cannot carry a multi-year retention claim on its own: lean on merged-PR review metadata — which persists with the repository indefinitely — as the primary approval evidence, and stream audit-log events into the evidence store (GitHub audit log streaming) if you want them retained alongside it. Same trap with CI: pipeline artifacts expire in weeks, so CI is the producer of eval records, and the eval-results table in the evidence store is where they survive.

The trace record that makes the top rows answerable:

```json
{
  "trace_id": "01J9F3T7Q8Z4",
  "tenant_id": "t_4821",
  "timestamp": "2026-09-02T14:31:07Z",
  "feature": "discharge-summary-draft",
  "model": {
    "provider": "anthropic",
    "id": "provider.model-id",
    "snapshot": "model-snapshot-id",
    "endpoint": "bedrock"
  },
  "prompt_template": {
    "id": "pt-a41f9c2",
    "repo": "acme/prompts",
    "committed_at": "2026-08-19T09:12:44Z"
  },
  "params": { "temperature": 0.2, "max_tokens": 1024 },
  "inputs": [
    {
      "name": "patient_record",
      "source": "records-api",
      "source_version": "v_20260902_1425",
      "hmac_sha256": "9f2b8c…",
      "tokens": 1893
    },
    {
      "name": "clinician_note",
      "source": "notes-api",
      "source_version": "v_20260902_1429",
      "hmac_sha256": "b71e04…",
      "tokens": 412
    }
  ],
  "output": {
    "sha256": "e4c1aa…",
    "tokens": 587,
    "finish_reason": "end_turn"
  },
  "latency_ms": 2140,
  "flags": {
    "safety_filter_triggered": false,
    "low_confidence": false,
    "guardrail_blocked": false
  },
  "decision": {
    "type": "human_review",
    "review_id": "rev-88f2",
    "outcome": "approved_with_edits"
  }
}
```

Every field is an identifier, a version, a count, a flag, or a hash. The model id is the one the platform actually invoked — here a Bedrock cross-region inference profile, which can route inference outside the calling region, which is why the endpoint field makes no region claim — so it joins directly against invocation logs and CloudTrail, exactly the cross-check an auditor runs; the snapshot field carries the normalized name for comparing across platforms. Nothing in the record is customer content, so it's designed to sit outside the PHI boundary and flow to the evidence store in ordinary observability retention — confirm the hashed-trace determination with your privacy officer, and never hash a small-value-space field unkeyed.

## Operating Notes

### A hash is only evidence while the source is durable

The keyed hash proves provenance only if the bytes it was computed over can be produced again. If your source systems mutate records in place, the hash of March's input won't match September's record and the proof evaporates. Record the source system's own version id next to the hash (as above), and confirm the source's retention meets your evidence retention. Where a source can't version, snapshot the input into the source system's audit trail at request time. That snapshot belongs to the system that already holds the data, not to your trace.

### Don't buy the observability platform before the discipline

The LLM-observability market (LangSmith, Langfuse, Braintrust, and a dozen others) defaults to capturing full prompts and completions, because that's what makes their debugging UX good. Pointed at a PHI-bearing feature with defaults on, they recreate the second losing hand as a SaaS sub-processor you now have to disclose. The tools are fine for eval orchestration and for content-off tracing, and self-hosting inside your boundary changes the calculus. But the compliance artifact is the content-free trace flowing to your own store, and that's 200 lines of middleware. Build that first; evaluate vendors after, with content capture treated as a data-flow decision rather than a default.

### The evidence is also your operations substrate

Records built for the auditor answer operational questions all week. Token counts by tenant and feature, multiplied by the vendor's price sheet, is your AI cost model. Latency and guardrail-flag rates by template SHA tell you whether Tuesday's prompt change caused Wednesday's pager noise. Review-decision ratios (edit rates creeping up, rejections clustering on one input source) catch quality drift between eval runs. Teams that treat AI auditability as pure compliance overhead build it grudgingly and thin. Treat it as the feature's flight recorder and it earns its keep monthly, with the audit answer as a byproduct.

## Frequently asked questions

**Q: How do you prove your AI features work to an auditor or enterprise buyer?**

You give them provenance and process, not content. Every question an assessor asks — what data the model saw, who approved an output, what was live on a given date, whether quality is holding — is answerable from metadata, version identifiers, and linked records, so no customer payloads ever need to leave your systems.

**Q: Should you log full prompts and completions for AI observability?**

Not for compliance evidence, especially when prompts are built from patient records or other sensitive data. Eighteen months of full prompts becomes an unplanned PHI repository with the wrong retention and access controls; log a content-free trace of ids, versions, hashes, and flags instead.

**Q: How can you prove what data went into a prompt without storing the prompt?**

Hash each variable input with keyed HMAC-SHA-256, using a key held outside the log pipeline, and record it next to the source system's name and version. Anyone holding the key can later pull the source record, rehash it, and match — proving input provenance without keeping a second copy of the data.

**Q: How do you show change management for AI prompts and models?**

Keep prompt templates in git, one file per template, changed only through reviewed PRs — the git log becomes your evidence for who changed a prompt, who approved it, and when it shipped. Pin dated model snapshot ids rather than aliases like 'latest', so you can always answer what was live on a given date.

**Q: How many eval cases do you need to prove AI quality isn't degrading?**

Fifty to two hundred de-identified or synthetic cases covering your feature's real distribution is a legitimate start; a thousand-case benchmark you never run is not. Run the set weekly by cron and in CI on every prompt or model change, retain the scores, and set regression thresholds that gate promotion.

**Q: Do you need a dedicated LLM-observability tool like LangSmith or Langfuse?**

Not for the compliance artifact. Those tools default to capturing full prompts and completions, which recreates the PHI-logging problem as a SaaS sub-processor you have to disclose; the content-free trace an auditor accepts is about 200 lines of middleware. They are fine for eval orchestration and content-off tracing, so build the trace first and evaluate vendors after.

Template: "LLM audit-evidence map + content-free trace schema" — branded PDF at https://kaansystems.com/templates/llm-feature-auditability.pdf

Source: https://kaansystems.com/library/llm-feature-auditability · Work with Kaan Systems: https://kaansystems.com/contact

---

# GitOps and Progressive Delivery with Automated Rollback

Canonical: https://kaansystems.com/library/gitops-progressive-delivery
Topic: Platform · Published: 2026-08-08 · Updated: 2026-10-12
Summary: CI pipelines that kubectl apply into production fail quietly: drift, rollback archaeology, cluster credentials living in your most-attacked system. A pull-based GitOps controller plus SLO-gated canaries fixes all three. The git history it produces is change-management evidence your auditor will accept without a single screenshot.

## Key takeaways

- Push-based deploys — CI running kubectl apply or helm upgrade against production with stored cluster credentials — decay into drift, rollback-by-archaeology, and cluster-admin access living in your most third-party-exposed system.
- GitOps inverts the write path: a controller like Argo CD or Flux runs inside the cluster, pulls from git, and continuously reconciles, so CI never holds cluster credentials and a compromised pipeline's blast radius drops to 'can open a PR.'
- Structure the deploy repo with app source split from desired state and environments as directories, not branches; environment branches diverge permanently after the first hotfix cherry-pick.
- Argo CD's syncPolicy has three switches — automated, prune, and selfHeal; give staging all three immediately but start prod on manual sync, flipping to automated only once the analysis gates have caught a real regression.
- An SLO-gated canary with Argo Rollouts shifts traffic in steps and runs AnalysisRuns against Prometheus, aborting to the untouched stable ReplicaSet in seconds when success rate or p99 latency breaches — no human, no CI run, no image pull.
- The git history this produces is change-management evidence auditors accept: every production change is a reviewed PR with an author, a branch-protection-enforced approver, a merge timestamp, and a commit SHA in Argo CD's sync history.

**What is GitOps with progressive delivery?** A controller inside the cluster pulls desired state from git and reconciles to match, so git becomes the only write path and CI loses its production credentials. Each change ships as an SLO-gated canary the machine reverts on breach, and the git history it produces is change-management evidence auditors accept.

## Why do fast deploys and change control fight?

Somewhere around the twentieth service, the deploy pipeline that got you here starts working against you. The pattern is common enough to have a smell: CI builds an image, then runs `kubectl apply` or `helm upgrade` against production using cluster credentials stored as pipeline secrets. Push-based deployment. It demos well in month one and decays quietly for the next two years.

The decay has a shape. First, drift: anything applied outside the pipeline (a hot-patched ConfigMap during an incident, a replica bump someone forgot to commit) persists forever, because nothing reconciles the cluster back to a declared state. Second, there is no authoritative record of desired state. The manifests in the repo describe what CI intended at some point in the past; the cluster is the only system that knows what is running now, and the cluster does not do code review. Third, rollback becomes archaeology. At 2 a.m., "roll back" means scrolling CI history, guessing which green build corresponds to the last good state, and re-running it while hoping nobody hand-edited anything since.

For a regulated SMB there is a fourth problem, and it is the one that surfaces in an audit. Your CI system holds credentials that write to production. CI is the most plugin-riddled, most third-party-exposed system you operate, and under this model it has cluster-admin. When the SOC 2 auditor asks how production changes are authorized, reviewed, and logged, the honest answer is "a pipeline job with a shared kubeconfig," followed by a week of screenshotting build logs. Change control exists as tribal ceremony, not as a property of the system.

## How does GitOps make rollback the audit trail?

Two moves, in order. First, invert the write path: nothing pushes to the cluster; a controller inside the cluster pulls from git, and git becomes the only way production changes. Second, make the deployment itself gradual and self-checking, so a bad change is caught by your SLOs and reversed by the machine before a human is paged.

```mermaid
flowchart TD
    PR["Pull request:<br/>reviewed, approved"] --> MERGE["Merge to main"]
    MERGE --> BUILD["CI builds image,<br/>pushes to ECR,<br/>bumps tag in deploy repo"]
    BUILD --> SYNC["Argo CD detects diff,<br/>syncs the Rollout"]
    SYNC --> W10["Canary: setWeight 10"]
    W10 --> A1["AnalysisRun:<br/>success rate + p99<br/>vs Prometheus"]
    A1 -->|pass| W50["setWeight 50"]
    A1 -->|fail| ABORT["Abort: shift all traffic to stable,<br/>scale canary to zero"]
    W50 --> A2["AnalysisRun repeats"]
    A2 -->|pass| W100["setWeight 100:<br/>canary promoted to stable"]
    A2 -->|fail| ABORT
    ABORT --> REVERT["Revert PR in deploy repo<br/>records the rollback"]
    classDef box fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
    classDef stage fill:#1F4A3E,stroke:#2E8B6F,stroke-width:1.5px,color:#E8F5EE;
    classDef dark fill:#0a1612,stroke:#2E8B6F,stroke-width:1.5px,color:#E8F5EE;
    class PR,MERGE,BUILD,SYNC box;
    class W10,W50,W100,A1,A2 stage;
    class ABORT,REVERT dark;
```

### Git as the only write path

GitOps is not "YAML in a repo." It is a specific inversion: an agent (Argo CD or Flux) runs inside the cluster, watches a git repository, and continuously reconciles the cluster toward what the repo declares. CI never touches the cluster. Its job shrinks to building images and updating an image tag in the deploy repo. Cluster credentials come out of CI entirely; the controller authenticates to git with a read-only deploy key, and the blast radius of a compromised pipeline drops from "production cluster" to "can open a PR."

The reconciliation loop is the point. It converts drift from slow rot into a visible, correctable event: the controller notices live state diverging from git and either flags it or reverts it. This is the same argument as [infrastructure drift detection](/library/infrastructure-drift-detection) applied at the workload layer, and it assumes the cluster fundamentals are already in place ([production-ready Kubernetes](/library/production-ready-kubernetes) covers that floor), with both sitting inside the broader [platform engineering foundation](/library/platform-engineering-for-regulated-startups) a regulated startup builds out. Rollback stops being archaeology: `git revert`, merge, converge. The last known good state is a commit SHA, not a memory.

### A deployment repo that scales

Split app source from desired state. App repos own code, tests, and the Dockerfile; a single deployment repo owns what runs where. The decoupling matters mechanically: CI in an app repo produces an image and opens a commit against the deploy repo bumping a tag. Deploy history and code history stay separate, which is exactly what you want when someone asks what changed in prod this week.

Inside the deploy repo, environments are directories, not branches. Environment branches feel natural and rot without exception. Promotion becomes merging `staging` into `prod`, which works until the first hotfix cherry-pick, after which the branches diverge permanently: merge conflicts in generated YAML, prod carrying changes staging never saw, and a history where comparing environments returns noise. With directories, promotion is a PR that copies a change from one overlay to another, and the difference between environments is a `diff -r` you can run any time.

```
deploy-repo/
├── apps/
│   └── api/
│       ├── base/                # shared manifests
│       └── overlays/
│           ├── staging/         # 2 replicas, shorter pauses
│           └── prod/            # 6 replicas, full canary
├── platform/                    # argo-rollouts, monitoring, ESO
└── argocd/                      # ApplicationSet + AppProject definitions
```

Kustomize keeps this honest: `base/` holds the manifests, overlays patch replica counts, analysis thresholds, and image tags. Bootstrap with an app-of-apps root Application or an ApplicationSet that generates one Argo CD Application per directory under `apps/`. Either way, adding a service to the platform is a directory and a PR, not a console session.

### Argo CD or Flux, in one paragraph

Argo CD gives you a UI that non-platform engineers use willingly, an RBAC model (AppProjects) that maps cleanly onto team and environment boundaries, and the Argo Rollouts ecosystem for progressive delivery. Flux is lighter, purely CRD-driven, more composable, and marginally easier to operate if nobody ever opens a dashboard. At 10 to 80 engineers the choice matters far less than the six weeks some teams spend debating it. Argo CD wins by default here because the canary machinery below is Argo Rollouts and the UI doubles as the manual sync gate. Pick one, record the decision, stop.

### Sync policy, term by term

Argo CD's `syncPolicy` has three switches that get cargo-culted together. `automated` means the controller applies changes as soon as they land in git, no human click. `prune` means resources deleted from git are deleted from the cluster; without it, removed manifests leave orphans running indefinitely, and an orphaned workload in a HIPAA environment is an unaccounted-for access path. `selfHeal` means manual edits to live objects are reverted to git state on the next reconcile, usually within minutes. That last one is drift correction in real time, and it will fight anyone who reaches for `kubectl edit`.

For early adoption: staging gets all three immediately. Prod starts with manual sync for the first month or two: drift and deleted-manifest orphans surface as OutOfSync in the UI (check the prune option when you click sync), but nothing is auto-corrected. The PR review remains the authorization; the sync button in the Argo UI is a timing control, letting you deploy inside a window rather than at merge time. Once the analysis gates below have caught a real regression and on-call trusts them, flip prod to `automated`, enabling `prune` and `selfHeal` at the same time. Manual sync forever is a smell: it means you never built confidence in your gates.

### Canary with an SLO gate

Rolling deployments verify that pods start. They say nothing about whether the new version is correct. Argo Rollouts replaces the Deployment with a Rollout resource that shifts traffic in steps and runs an AnalysisRun between steps, querying Prometheus against your actual SLOs.

The sequence that works at this scale: shift 10 percent of traffic to the canary, pause five minutes, run analysis (success rate and p99 latency over the canary pods), then 50 percent, analyze again, then 100. If any AnalysisRun breaches its `failureLimit`, Rollouts aborts: traffic shifts entirely back to the stable ReplicaSet, which was never scaled down, and the canary scales to zero. Rollback takes seconds and requires no human, no CI run, no image pull. The failed AnalysisRun results and the aborted revision's pod template stay inspectable for diagnosis instead of being stomped by a redeploy.

Honest constraints. You need real traffic shifting: ALB weighted target groups via ingress annotations, or a mesh; plain Service-based weighting is pod-count-granular and lumpy. And you need enough traffic for the statistics to mean something. At 5 requests per second, a 10 percent canary sees half a request per second: roughly 150 requests over a five-minute window, and about 60 per two-minute query window — workable for success rate, meaningless for p99. Low-traffic internal services should run a simpler two-step canary with longer pauses, or skip canary for blue-green. Do not fake statistical rigor where there is no data.

### The audit artifact you get for free

Under this model, every production change is a PR with a named author, a required approver enforced by branch protection, a passing status check, a merge timestamp, and a deterministic link to what the cluster ran (the commit SHA in Argo CD's sync history). That is change management in the sense auditors mean it, produced as a side effect of shipping. Evidence collection becomes a query over the git log of `overlays/prod` joined with the Argo CD API, feedable into an [evidence pipeline](/library/evidence-pipelines-101) instead of a quarterly screenshot hunt. Keep [one-page ADRs](/library/one-page-adrs) in the same repo under `docs/adr/` so the reasoning behind canary thresholds and sync policy lives next to the config it explains. Auditors read those too, and it beats reconstructing intent from Slack.

## The Template

Deployment-repo layout, a production Rollout with a three-stage canary, and the AnalysisTemplate that gates it. Tune the thresholds to your SLOs, not the other way around.

```
deploy-repo/
├── bootstrap/
│   └── root-app.yaml                  # single Application pointing at argocd/
├── argocd/
│   ├── applicationset.yaml            # one Application per apps/*/overlays/*
│   └── appproject-prod.yaml           # RBAC: who may sync prod
├── platform/
│   ├── argo-rollouts/                 # controller install, version pinned
│   ├── external-secrets/              # ESO + ClusterSecretStore (AWS SM)
│   └── monitoring/                    # Prometheus, ServiceMonitors
├── apps/
│   └── api/
│       ├── base/
│       │   ├── kustomization.yaml
│       │   ├── rollout.yaml           # below
│       │   ├── service-stable.yaml
│       │   ├── service-canary.yaml
│       │   ├── ingress.yaml           # ALB, weighted target groups
│       │   └── analysis-template.yaml # below
│       └── overlays/
│           ├── staging/
│           │   └── kustomization.yaml # replicas 2, shorter pauses
│           └── prod/
│               └── kustomization.yaml # replicas 6, image tag pinned here
└── docs/
    └── adr/
        └── 0007-canary-slo-thresholds.md
```

```yaml
# apps/api/base/rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api
spec:
  replicas: 6
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app: api
  strategy:
    canary:
      stableService: api-stable
      canaryService: api-canary
      trafficRouting:
        alb:
          ingress: api
          servicePort: 8080
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: api-slo-check
            args:
              - name: service
                value: api-canary
        - setWeight: 50
        - pause: { duration: 10m }
        - analysis:
            templates:
              - templateName: api-slo-check
            args:
              - name: service
                value: api-canary
        - setWeight: 100
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          # tag is set by CI via `kustomize edit set image` in overlays/prod
          image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/api:sha-0000000
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 5
          resources:
            requests: { cpu: 250m, memory: 256Mi }
            limits: { memory: 512Mi }
```

```yaml
# apps/api/base/analysis-template.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: api-slo-check
spec:
  args:
    - name: service
  metrics:
    - name: success-rate
      interval: 60s
      count: 5
      # non-5xx share of requests must stay at or above 99%
      successCondition: len(result) > 0 && result[0] >= 0.99
      failureLimit: 2            # tolerates two failed measurements; a third aborts
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc.cluster.local:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service}}",code!~"5.."}[2m]))
            /
            sum(rate(http_requests_total{service="{{args.service}}"}[2m]))
    - name: p99-latency
      interval: 60s
      count: 5
      successCondition: len(result) > 0 && result[0] <= 0.5
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc.cluster.local:9090
          query: |
            histogram_quantile(0.99,
              sum(rate(http_request_duration_seconds_bucket{service="{{args.service}}"}[2m])) by (le)
            )
```

On abort, Rollouts shifts all traffic to stable and scales the canary down; the revert PR in the deploy repo is the durable record of the rollback. Wire Argo CD Notifications to Slack so an aborted rollout pings the service owner, not just the platform channel.

## Operating Notes

### Tag bumps are a machine's job

The commit that changes an image tag in the deploy repo should be opened by CI: a bot PR for prod, a direct commit for staging. Argo CD Image Updater can do this, but a 20-line CI job that runs `kustomize edit set image` and opens a PR is easier to audit and easier to debug. Humans review promotions; they should not hand-type tags, because hand-typed tags are where typos meet production. The bot PR also gives your prod promotion the same approval trail as every other change, which is the entire premise.

### Secrets never enter the deploy repo

A GitOps repo is read widely by design: every engineer, CI, the controller. Kubernetes Secrets, even base64-encoded, do not belong in it. Run External Secrets Operator pointed at AWS Secrets Manager and commit only ExternalSecret references. Secret values then carry their own audit trail in CloudTrail, separate from config history, which is the separation of duties your auditor expects to find anyway. If a secret does land in git, treat it as leaked and rotate; history rewrites are not remediation.

### The first month is an argument with selfHeal

Enabling `selfHeal` surfaces every hand-edit habit your team has, usually within days, usually mid-debugging when someone's live tweak vanishes under them. This is the system working. Announce the cutover, document the break-glass path (disable selfHeal on a single Application, annotate who and why, restore within 24 hours), and hold the line. Teams that carve out permanent exceptions end up operating two systems: GitOps for the services that behave, and vibes for the ones that page.

## Frequently asked questions

**Q: What is GitOps?**

GitOps is a specific inversion of the deploy path: an agent like Argo CD or Flux runs inside the cluster, watches a git repository, and continuously reconciles the cluster toward what the repo declares. CI never touches the cluster — its job shrinks to building images and bumping an image tag in the deploy repo, and cluster credentials leave CI entirely.

**Q: Should I use Argo CD or Flux?**

At 10 to 80 engineers the choice matters far less than the weeks teams spend debating it. Argo CD gives you a UI non-platform engineers use willingly, an RBAC model that maps to team and environment boundaries, and the Argo Rollouts ecosystem for canaries; Flux is lighter and purely CRD-driven. Pick one, record the decision, and move on.

**Q: How does automated rollback work in progressive delivery?**

Argo Rollouts replaces the Deployment with a Rollout that shifts traffic in steps and runs an AnalysisRun against Prometheus between them. If any AnalysisRun breaches its failureLimit, Rollouts aborts: all traffic shifts back to the stable ReplicaSet, which was never scaled down, and the canary scales to zero. Rollback takes seconds and needs no human, no CI run, and no image pull.

**Q: Should environments be git branches or directories?**

Directories. Environment branches feel natural but rot: after the first hotfix cherry-pick they diverge permanently, producing merge conflicts in generated YAML and prod carrying changes staging never saw. With directories, promotion is a PR that copies a change from one overlay to another, and comparing environments is a diff you can run any time.

**Q: Do I need a service mesh for canary deployments?**

Not necessarily, but you need real traffic shifting — ALB weighted target groups via ingress annotations, or a mesh. Plain Service-based weighting is pod-count-granular and lumpy. You also need enough traffic for the statistics to mean something; low-traffic internal services should run a simpler two-step canary with longer pauses or use blue-green instead.

**Q: How does GitOps help with SOC 2 or audit change management?**

Every production change becomes a PR with a named author, a required approver enforced by branch protection, a passing status check, a merge timestamp, and a deterministic link to what the cluster ran via the commit SHA in Argo CD's sync history. That is change management as auditors mean it, produced as a side effect of shipping — evidence collection becomes a query over the git log joined with the Argo CD API.

Template: "GitOps Deploy Repo + SLO-Gated Canary" — branded PDF at https://kaansystems.com/templates/gitops-progressive-delivery.pdf

Source: https://kaansystems.com/library/gitops-progressive-delivery · Work with Kaan Systems: https://kaansystems.com/contact

---

# Killing Long-Lived Credentials with Workload Identity and Dynamic Secrets

Canonical: https://kaansystems.com/library/killing-long-lived-credentials
Topic: Platform · Published: 2026-08-08 · Updated: 2026-10-12
Summary: Static credentials are the breach vector that actually gets used: they work until someone revokes them, and nobody revokes them. A phased plan for replacing IAM user keys, CI secrets, and database passwords with workload identity, OIDC federation, and dynamic secrets, with the static-credential count as the program KPI.

## Key takeaways

- Static credentials are the breach vector attackers actually use: a stolen key keeps working until a human revokes it, and credential-based intrusions have averaged around 240 days to identify.
- Inventory comes first — `aws iam generate-credential-report` plus a gitleaks or trufflehog scan across full git history builds a register whose row count becomes the program KPI, published weekly.
- Give workloads identity instead of stored keys: IRSA or EKS Pod Identity issues hour-long credentials that live in process memory, with nothing on disk, in a Kubernetes Secret, or in etcd to steal.
- In CI/CD, replace GitHub repository secrets with OIDC federation and pin the trust policy's `sub` claim to a specific repo and ref; a `repo:your-org/*` wildcard lets any workflow in the org assume your deploy role.
- For human access, put IAM Identity Center in front of your existing IdP and then delete every IAM user, keeping only two break-glass users with offline hardware-MFA keys wired to a CloudTrail alarm.
- Migrate one credential class at a time — inventory, build the new path, run both in parallel, then revoke — and accept the asymptote: the durable end state is break-glass plus the handful of vendors that only support API keys.

**How do you kill long-lived credentials?** Stop storing secrets and issue short-lived ones on proof of identity: workload identity (IRSA or EKS Pod Identity) for pods, OIDC federation for CI, SSO sessions for humans, and RDS IAM auth or rotating secrets for databases. Around 90% of a typical AWS estate can drop static keys entirely.

The phased plan assumes a real account boundary. If you do not have one yet, start with a [multi-account landing zone](/library/multi-account-aws-landing-zone). The [7-day Assessment](/assessment) includes the IAM credential report review.

## Why do long-lived keys survive every rotation reminder?

An attacker who wants into your AWS account does not need a zero-day. They need one of the several hundred static credentials your organization has minted over the years, and they only need it once. GitHub is scraped for `AKIA` prefixes continuously; a canary access key pushed to a public repo receives its first unauthorized API call in under ten minutes. The CircleCI breach in January 2023 worked this way at one remove: an engineer's laptop was compromised, a session was hijacked, and every customer who had stored AWS keys, deploy tokens, or signing certificates in CircleCI got the worst email of the quarter. Rotate everything, now, and hope your rotation runbooks exist.

The property that makes static credentials the dominant real-world breach vector is simple: stolen does not mean expired. A short-lived credential lifted from a compromised pod is worth an hour to an attacker. A long-lived one is worth however many months pass before someone notices, and breach-cost studies have put mean identification time for credential-based intrusions at around 240 days for years running. The secret in your CI settings, the IAM user key sitting in a developer's `~/.aws/credentials` since 2021, the database password in a Helm values file: each one works until a human revokes it, and revocation requires the human to know it was taken.

For a regulated SMB the problem doubles. HIPAA and SOC 2 both push you toward credential rotation, so teams perform quarterly rotation theater: a spreadsheet, a painful afternoon, and a couple of production incidents per year caused by the rotation itself. The exercise mostly proves that the process is manual and feared. The better answer is structural. Stop storing credentials at all, wherever a workload, a pipeline, or a human can prove its identity and be issued a short-lived credential on demand. For roughly 90% of a typical AWS estate, they can.

## How do you replace them with workload identity?

The organizing principle: replace possession of a secret with proof of identity at every boundary. AWS already operates the exchange desk (STS). Your job is to register the identity providers and delete the static keys. The trust flow for a Kubernetes workload looks like this; the CI and human flows are the same shape with a different token issuer.

```mermaid
flowchart LR
    subgraph EKS["EKS cluster"]
        POD["payments-api pod<br/>projected SA token"]
    end
    ISSUER["Cluster OIDC issuer<br/>signs JWT with sub claim<br/>system:serviceaccount:prod:payments-api"]
    STS["AWS STS<br/>AssumeRoleWithWebIdentity"]
    TRUST["IAM role trust policy<br/>issuer and sub must match"]
    CREDS["Temporary credentials<br/>1h TTL, auto-refreshed<br/>never written to disk"]
    API["AWS APIs<br/>S3, SQS, KMS"]

    POD -->|"presents JWT"| STS
    ISSUER -.->|"publishes signing keys"| STS
    STS -->|"validates against"| TRUST
    TRUST -->|"issues"| CREDS
    CREDS -->|"signs requests to"| API

    classDef box fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
    classDef hot fill:#1F4A3E,stroke:#2E8B6F,stroke-width:1.5px,color:#E8F5EE;
    classDef grp fill:#0a1612,stroke:#2E8B6F,color:#E8F5EE;
    class POD,ISSUER,STS,TRUST,API box;
    class CREDS hot;
    class EKS grp;
```

### The inventory: where static credentials hide

Counting comes first. `aws iam generate-credential-report` lists every IAM user access key with its age and last-used date; on a typical 30-engineer estate that report shows 40 to 80 keys, a third of them unused in the last 90 days. Then the places the report cannot see: repository and organization secrets in GitHub Actions, database passwords in ECS task definitions and Helm values, kubeconfigs with embedded client certificates that never expire, service-account JSON files, and the vendor API key living in a founder's dotfiles because it predates the secrets manager. Run gitleaks or trufflehog across full git history, not just HEAD; the file someone deleted in 2023 is still in the object store. The output is a register: credential, system, owner, blast radius if leaked, last-used date. The row count is the KPI for the entire program, and every phase below ends by making it smaller.

### Workloads: identity instead of storage

IRSA (IAM Roles for Service Accounts) is the established path on EKS. Each cluster ships an OIDC issuer; you register it in IAM once, then write a role trust policy that matches the token's `sub` claim to a specific service account, such as `system:serviceaccount:prod:payments-api`. The pod mounts a projected, auto-rotated token, and the AWS SDK exchanges it via `AssumeRoleWithWebIdentity` with no application changes. Credentials live in process memory, last an hour, and refresh themselves. Nothing on disk, nothing in a Kubernetes Secret, nothing in etcd to exfiltrate.

EKS Pod Identity (GA since late 2023) reaches the same end with less ceremony: no per-cluster OIDC provider registration, an agent handling `AssumeRoleForPodIdentity`, and role associations managed through a plain EKS API call. That matters once you run three or four clusters and tire of trust policies referencing cluster-specific issuer URLs. The trade-offs are minor but real: Pod Identity does not cover Fargate pods, and cross-account access needs explicit role chaining where IRSA can trust directly. Pick one mechanism per cluster and stay consistent. On GKE the same design is called Workload Identity Federation; on AKS, Entra Workload ID. Same JWT, same exchange.

### CI/CD: federation, not repository secrets

The `AWS_ACCESS_KEY_ID` stored in GitHub repository secrets is usually the highest-blast-radius static credential an SMB owns: it carries deploy-level permissions and is exposed to every workflow, third-party action, and maintainer account. GitHub Actions issues OIDC tokens natively. Register `token.actions.githubusercontent.com` as an identity provider in IAM and the pipeline assumes a role directly, so the next CircleCI-class vendor breach is someone else's rotation emergency.

The trust policy conditions do the security work (full JSON in the template below): `aud` pinned to `sts.amazonaws.com`, and `sub` pinned to `repo:your-org/your-repo:ref:refs/heads/main`. The classic misconfiguration is a `StringLike` condition on `repo:your-org/*`, which lets any workflow in any repository in the org — including a pull request against some archived experiment — assume your deploy role. Security researchers have found `repo:*` in the wild, which is an open door for every GitHub account on the planet. Pin to repository and ref, and give pull-request builds a separate role that can run `terraform plan` and nothing else.

### Humans: SSO sessions, zero IAM users

Put IAM Identity Center in front of whatever IdP you already pay for (Google Workspace, Okta, Entra). Engineers get permission sets, sessions of 8 to 12 hours, and `aws sso login` for the CLI. Then the uncomfortable part: delete the IAM users. All of them. The end state is a credential report with zero rows, plus two break-glass IAM users with hardware MFA whose access keys exist only on paper in a safe, wired to a CloudTrail alarm that pages on any use. This is Phase 1 of [zero trust for SMBs](/library/zero-trust-for-smbs) applied to the AWS control plane: identity-based, short-lived, centrally revocable. When someone leaves, disabling one IdP account kills every session within hours. Compare that with the offboarding checklist line that reads "rotate any keys they may have had."

### Databases and third-party APIs

RDS IAM authentication swaps the password for a 15-minute connection token, on Postgres, MySQL, and MariaDB engines including Aurora. It fits services with modest connection churn and works cleanly behind RDS Proxy. It does not fit everything: IAM-auth connection establishment carries extra overhead on the database side — token verification and mandatory TLS — and AWS recommends it only below roughly 200 new connections per second; some ORMs fight the auth plugin, and per-user tokens map awkwardly onto shared connection pools without a proxy in the path.

Where a password must exist, make it self-rotating. Secrets Manager with the managed Lambda rotators handles RDS credentials on a schedule using the alternating-users strategy, and applications that fetch the secret at connect time never notice a rotation. Vault's dynamic database secrets go further: per-lease database users created on demand with a TTL, giving every workload its own expiring identity inside the database. Honest guidance at 10-80 engineers: Vault is a distributed system you have to operate (HA, unseal, upgrades, policy sprawl), and running it solely for database credentials is overkill when Secrets Manager rotation delivers most of the value with none of the pager load. Vault earns its keep when you are multi-cloud, need dynamic credentials across many engines, or already run it for something else. HCP Vault splits the difference if lease-based credentials are genuinely required without the ops burden.

### The migration: parallel run, then revoke

Four moves per credential class: inventory, build the new issuance path, run both in parallel, revoke the old one. Deactivation is the safety net. IAM access keys support an Inactive state, so flip a key inactive for a week before deleting it and watch CloudTrail for whatever screams. Last-used data sets the sequence: dead keys first (instant wins, zero risk), then CI, then workloads, then humans. Publish the static-credential count weekly. A graph that reads 74, 61, 40, 19 does more for program momentum than any policy document.

Then the residue: the three vendors that only support API keys. There are always three. Scope each key to the minimum the integration needs, store it in Secrets Manager, put a 90-day rotation cadence on the calendar, assign a named owner, and alert on usage anomalies. Those keys become line items in your [vendor risk scoring](/library/vendor-risk-scoring), because their rotation story is now part of the vendor's risk posture, not only yours.

## The Template

The migration checklist below is phased so that each phase ends with a revocation, not a deployment. A phase is not done while the old credential still works.

```markdown
# No-Long-Lived-Credentials Migration

## Phase 0: Inventory (week 1)
- [ ] Run `aws iam generate-credential-report`; export all access keys with age and last-used
- [ ] Enumerate CI secrets: GitHub org, repo, and environment secrets
- [ ] Grep deploy artifacts for embedded creds: task definitions, Helm values, compose files, .env
- [ ] gitleaks/trufflehog across full git history of all active repos
- [ ] List kubeconfigs with embedded client certs; note expiry (often: none)
- [ ] Build the register: credential | system | owner | blast radius | last used
- [ ] Record the baseline count; this number is the program KPI

## Phase 1: Workloads (weeks 2-4)
- [ ] Register cluster OIDC provider in IAM (IRSA) or enable the EKS Pod Identity agent
- [ ] One IAM role per service; trust pinned to namespace + service account
- [ ] Migrate services off node-role and env-var creds; verify AssumeRoleWithWebIdentity in CloudTrail
- [ ] Delete AWS credentials from Kubernetes Secrets and task-definition env vars

## Phase 2: CI/CD (weeks 3-5)
- [ ] Create the token.actions.githubusercontent.com OIDC provider in IAM
- [ ] Deploy role trust pinned to repo + branch (JSON below); separate plan-only role for PRs
- [ ] Cut workflows over to aws-actions/configure-aws-credentials with role-to-assume
- [ ] Deactivate, then delete, the CI IAM user and its stored repo secrets

## Phase 3: Humans (weeks 4-6)
- [ ] IAM Identity Center connected to the IdP; permission sets replace per-user policies
- [ ] CLI access via `aws sso login`; session duration 12h or less
- [ ] Create 2 break-glass IAM users: hardware MFA, keys stored offline, CloudTrail alarm on use
- [ ] Flip all remaining IAM user keys Inactive for 7 days, then delete the users

## Phase 4: Data stores (weeks 5-8)
- [ ] RDS IAM auth for low-churn services (Postgres/MySQL/Aurora)
- [ ] Secrets Manager rotation (managed Lambda, alternating users) for the rest
- [ ] Apps fetch DB credentials at connect time from Secrets Manager, never from env vars
- [ ] Rotate every DB password once through the new path to invalidate leaked history

## Phase 5: Third parties (ongoing)
- [ ] Per vendor key: minimum scope, Secrets Manager entry, named owner, 90-day rotation
- [ ] Usage alerting per key (API audit log or billing anomaly)
- [ ] Fold key-rotation posture into vendor risk reviews

## Steady state
- [ ] Static-credential count published weekly (target: break-glass plus vendor keys, nothing else)
- [ ] CI check blocking AKIA* and other known key formats in commits
- [ ] SCP denying iam:CreateUser and iam:CreateAccessKey outside the break-glass path (SCPs never bind the Organizations management account, so this holds only if workloads live in member accounts; single-account estates: explicit deny in a permissions boundary plus a CloudTrail/EventBridge alarm on CreateUser/CreateAccessKey)
```

The GitHub Actions trust policy, with the conditions that matter:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::111122223333:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:sub": "repo:your-org/your-repo:ref:refs/heads/main"
        }
      }
    }
  ]
}
```

Never widen the `sub` condition to `repo:your-org/*` on the deploy role. If several repos genuinely need deploy access, give each its own statement or its own role.

## Operating Notes

### Revocation is the real test

If revoking a particular credential feels frightening, you have located a system whose ownership and issuance path nobody understands. That fear is the finding; write it down before you fix it. The parallel-run pattern exists to convert fear into boredom: the new path serves traffic for a week while the old credential sits deactivated but recoverable. Apply the same discipline to break-glass. Twice a year, actually open the safe, actually use the offline key, and confirm the alarm pages someone. A break-glass procedure that has never been exercised is a decorative one.

### The audit trail arrives for free

Every `AssumeRole` and `AssumeRoleWithWebIdentity` call is a CloudTrail event binding an action to an identity, a session, and a timestamp. Once credentials are issued rather than stored, your rotation control is satisfied structurally: the auditor question "how do you rotate credentials?" gets the answer "they expire hourly, and issuance is logged." That is a materially stronger position for SOC 2 CC6.1 than a spreadsheet of quarterly rotations, and it feeds directly into [evidence pipelines](/library/evidence-pipelines-101), because the proof is a queryable log rather than a screenshot someone remembers to take.

### Don't chase the last three

The static-credential count follows an asymptote. It falls fast through the dead keys and CI secrets, slower through data stores, and then lands on a small stubborn number: break-glass plus the vendors that will not do OIDC. Accept the asymptote and guard the perimeter instead. An SCP denying new access-key creation means the count cannot silently climb back up, and a CI secret-format check means the next leaked key is caught at commit time. The durable win is cultural: engineers who joined after the migration have never configured a static AWS key and would not know where to put one. At that point the migration has stopped being a project and become the default, which is the only finish line that holds.

## Frequently asked questions

**Q: Why are long-lived credentials such a security risk?**

Because stolen does not mean expired: a static key keeps working until a human notices and revokes it, and credential-based intrusions have averaged around 240 days to identify. A short-lived credential lifted from a compromised pod is worth an hour to an attacker; a long-lived one is worth however many months pass before someone catches it.

**Q: How do I replace AWS access keys in my GitHub Actions pipeline?**

Register `token.actions.githubusercontent.com` as an IAM identity provider and have the pipeline assume a role directly via GitHub's native OIDC token, instead of storing `AWS_ACCESS_KEY_ID` in repository secrets. Pin the trust policy's `sub` to a specific repo and ref rather than a `repo:your-org/*` wildcard, and give pull-request builds a separate plan-only role.

**Q: What is the difference between IRSA and EKS Pod Identity?**

Both give pods short-lived credentials without stored keys, but IRSA registers a per-cluster OIDC provider in IAM and matches a role's trust policy to the token's `sub` claim, while EKS Pod Identity (GA since late 2023) skips that registration and manages role associations through a plain EKS API call — easier once you run several clusters. The trade-offs: Pod Identity does not cover Fargate pods and needs explicit role chaining for cross-account access, so pick one mechanism per cluster and stay consistent.

**Q: Do I need HashiCorp Vault to get rid of database passwords?**

Usually not. RDS IAM authentication swaps the password for a 15-minute connection token on Postgres, MySQL, and Aurora, and Secrets Manager with managed Lambda rotators handles the rest on a schedule. Vault's dynamic per-lease database users go further, but at 10-80 engineers running Vault solely for database credentials is overkill — it earns its keep when you are multi-cloud or already operate it for something else.

**Q: How do I migrate off static credentials without breaking production?**

Work one credential class at a time in four moves: inventory, build the new issuance path, run both in parallel, then revoke the old credential. Deactivation is the safety net — flip an IAM key Inactive for a week and watch CloudTrail before deleting it. Sequence by last-used data: dead keys first, then CI, then workloads, then humans.

**Q: Does killing long-lived credentials help with SOC 2 compliance?**

Yes. Once credentials are issued rather than stored, every AssumeRole call is a CloudTrail event binding an action to an identity, a session, and a timestamp, so your rotation control is satisfied structurally — they expire hourly and issuance is logged. That is a materially stronger position for SOC 2 CC6.1 than a spreadsheet of quarterly rotations, and it feeds evidence pipelines directly.

Template: "No-Long-Lived-Credentials Migration Checklist" — branded PDF at https://kaansystems.com/templates/killing-long-lived-credentials.pdf

Source: https://kaansystems.com/library/killing-long-lived-credentials · Work with Kaan Systems: https://kaansystems.com/contact

---

# Building a Multi-Account AWS Landing Zone

Canonical: https://kaansystems.com/library/multi-account-aws-landing-zone
Topic: Platform · Published: 2026-08-08 · Updated: 2026-10-12
Summary: One AWS account with tag-based separation concentrates blast radius and fails audits. A landing zone design for 10-80 engineer teams: OU topology, service control policies, identity, networking restraint, and the Control Tower decision, with a reference template.

## Key takeaways

- A single AWS account separated only by tags concentrates blast radius, shares account-scoped quotas like Lambda concurrency and EC2 vCPU, and fails the environment-isolation question SOC 2 and HIPAA assessors ask.
- The fix is AWS Organizations: one account per isolation boundary, grouped into five OUs and roughly ten accounts, constrained by service control policies.
- Service control policies grant nothing — they are a deny-list ceiling layered on top of IAM, and they never apply to the management account, which is why nothing should run there.
- For a 10-80 engineer regulated SMB that already runs Terraform through CI, roll your own landing zone in plain Terraform and skip Control Tower — and don't adopt AFT at this scale.
- Use IAM Identity Center from day one with SCIM sync and zero per-account IAM users, so a quarterly access review becomes an export of assignments instead of an archaeology dig.
- Retrofit under a live workload without a migration weekend: invite the existing account in as prod, build the other accounts around it, and apply SCPs to the staging OU before prod.

**What is an AWS landing zone, and how should a regulated SMB build one?** A landing zone is a multi-account AWS Organizations structure: one account per isolation boundary, grouped into five OUs and ten accounts, constrained by service control policies and unified by IAM Identity Center. Get it approximately right at 20 engineers rather than untangling one account at 60.

## Why does a single AWS account stop being enough?

The single-account AWS estate is the default outcome of a startup that shipped fast. Production, staging, the CI runners, a data engineer's Redshift experiment, and an abandoned marketing microsite all live in account 481516234200, separated by naming prefixes and a tagging policy that was enforced for about six weeks in 2023. The security boundary between staging and production is IAM policy syntax. Somewhere in the 140 roles that have accumulated, at least one carries a `Resource: "*"` nobody has audited, and a compromised staging workload can reach production database snapshots through it.

The damage shows up on three fronts. Security: blast radius. A leaked CI token hands an attacker the entire company, because the account is the only hard wall AWS gives you and everything lives inside it. Operations: shared quotas. The load test that exhausted Lambda concurrency took production down with it, and the same goes for EC2 vCPU limits, EIPs, and API rate limits, all of which are account-scoped. Audit: the SOC 2 assessor asks how production is isolated from development, and the honest answer is "a tag." For a HIPAA workload that answer gets expensive, because environment separation, access minimization, and tamper-evident logging are things assessors expect to see structurally, not procedurally.

The fix has been stable for years: AWS Organizations, one account per isolation boundary, grouped into OUs, constrained by service control policies. What keeps teams at 15 to 40 engineers from doing it is the memory of what multi-account used to mean: credential juggling, thirteen browser profiles, cross-account confusion. That era ended. IAM Identity Center makes many accounts feel like one login, and Organizations makes accounts nearly free to create. The remaining cost is design: which accounts, which OUs, which guardrails. That design is the landing zone, and getting it approximately right at 20 engineers is dramatically cheaper than untangling one account at 60. It is the foundational layer of the broader [platform engineering practice](/library/platform-engineering-for-regulated-startups) a regulated startup builds out as it grows.

## What does a minimum multi-account landing zone include?

Five OUs, roughly ten accounts. The diagram is the architecture; the subsections argue for it.

```mermaid
flowchart TD
    ROOT["Management account<br/>billing + org admin<br/>nothing runs here"]
    SEC["Security OU"]
    INFRA["Infrastructure OU"]
    WORK["Workloads OU"]
    SAND["Sandbox OU"]
    LOG["log-archive<br/>org CloudTrail, Object Lock<br/>no human logins"]
    AUD["security-tooling<br/>GuardDuty, Security Hub<br/>delegated admin"]
    NET["network<br/>peering/TGW, IPAM, egress"]
    SHARED["shared-services<br/>CI runners, ECR, tooling"]
    PROD["prod"]
    STAGE["staging"]
    SBX["sandbox-*<br/>budget-capped, nuked nightly"]

    ROOT --> SEC
    ROOT --> INFRA
    ROOT --> WORK
    ROOT --> SAND
    SEC --> LOG
    SEC --> AUD
    INFRA --> NET
    INFRA --> SHARED
    WORK --> PROD
    WORK --> STAGE
    SAND --> SBX

    classDef mgmt fill:#1F4A3E,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
    classDef ou fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
    classDef acct fill:#0a1612,stroke:#2E8B6F,stroke-width:1.5px,color:#E8F5EE;
    class ROOT mgmt;
    class SEC,INFRA,WORK,SAND ou;
    class LOG,AUD,NET,SHARED,PROD,STAGE,SBX acct;
```

### One account per blast radius

A VPC separates networks. An account separates everything: IAM principals, S3 buckets, KMS keys, Lambda functions, DynamoDB tables, service quotas, billing, and the scope of a stolen credential. A large fraction of AWS never touches a VPC at all, which is exactly why VPC-only separation fails quietly. Your "isolated" staging environment shares an IAM universe with production, and one over-permissive role is a bridge. Between accounts, every access path must be explicit: a resource policy or an assumed role, both of which are visible, auditable, and revocable. "Who can touch production" stops being a policy-analysis research project and becomes a list of principals in one account. That is a sentence you can put in front of an auditor.

The account roles, briefly. The **management account** runs nothing: no VPCs, no workloads, no data buckets. SCPs do not apply to it (more below), so anything running there is ungoverned by definition. It exists for billing and org administration, and its root credentials live in a safe. The **Security OU** holds two accounts. The log-archive account receives the organization CloudTrail and Config history into an S3 bucket with Object Lock in compliance mode; nobody logs in here, and even an administrator cannot delete a log object before its retention expires. The security-tooling account is delegated administrator for GuardDuty, Security Hub, and IAM Access Analyzer, and is where a security engineer actually works. The **Infrastructure OU** holds the network account (connectivity, IPAM, egress) and shared-services (CI runners, ECR, internal tooling). The **Workloads OU** holds prod and staging as separate accounts, not separate VPCs, so that promotion through environments exercises the same IaC against genuinely identical topology. The **Sandbox OU** is where the freedom you removed from production goes: per-engineer accounts with a $200 budget alarm, no connectivity to anything, wiped nightly with aws-nuke.

That is ten accounts. Resist the urge to mint one per microservice; at 10-80 engineers, account sprawl recreates the original governance problem with extra steps.

### Service control policies: a deny-list, not permissions

SCPs are the most misunderstood piece of Organizations. They grant nothing. They are a ceiling: an action is allowed only if IAM policy grants it AND no SCP denies it. The default `FullAWSAccess` policy stays attached everywhere, and you layer denies on top. And they never apply to the management account, which is the technical reason nothing may run there. The guardrails you spent a week designing simply do not exist in that account.

A starter set that covers most of the risk for a regulated SMB: deny leaving the organization, deny tampering with CloudTrail, deny root-user actions, restrict regions to the ones you operate in, and protect the log-archive bucket's configuration. The first three in one policy, attached at the root:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyLeavingOrg",
      "Effect": "Deny",
      "Action": "organizations:LeaveOrganization",
      "Resource": "*"
    },
    {
      "Sid": "DenyCloudTrailTamper",
      "Effect": "Deny",
      "Action": [
        "cloudtrail:StopLogging",
        "cloudtrail:DeleteTrail",
        "cloudtrail:UpdateTrail",
        "cloudtrail:PutEventSelectors"
      ],
      "Resource": "*"
    },
    {
      "Sid": "DenyRootUser",
      "Effect": "Deny",
      "Action": "*",
      "Resource": "*",
      "Condition": {
        "StringLike": { "aws:PrincipalArn": "arn:aws:iam::*:root" }
      }
    }
  ]
}
```

Two operational cautions. First, stage every SCP through a low-stakes OU before it reaches Workloads; the AccessDenied error message now names the offending SCP, but debugging inheritance across nested OUs is still tedious at 5 p.m. on a Friday. Second, mind the 5-policies-per-OU and quota-limited document size; consolidate statements rather than attaching a policy per rule.

### Control Tower or Terraform

Control Tower gives you a working landing zone in an afternoon: org CloudTrail, Config baseline, a curated guardrail catalog, and Account Factory for vending accounts. The costs arrive later. It is opinionated about OU structure, its guardrails drift-check against their own model rather than yours, landing zone version upgrades are an event, and Account Factory provisions through Service Catalog, which is slow and awkward to drive from automation. AFT (Account Factory for Terraform) exists to fix that last problem and brings four repositories, CodePipeline, CodeBuild, and a DynamoDB table of its own. At this scale, AFT tends to be the worst of both worlds: Control Tower's rigidity plus a bespoke pipeline someone must maintain.

Rolling your own means `aws_organizations_organization`, `aws_organizations_organizational_unit`, `aws_organizations_account`, and `aws_organizations_policy` in plain Terraform, and owning the baseline yourself: the org trail, the Config aggregator, GuardDuty org enrollment, Identity Center assignments. More upkeep, full control, and the entire org becomes a single `terraform plan` whose drift your existing [drift detection](/library/infrastructure-drift-detection) tooling already watches. The org definition for a ten-account estate is a few hundred lines.

The recommendation for a 10-80 engineer regulated SMB: if you already run Terraform through CI and one engineer owns platform, roll your own and skip Control Tower entirely. The org config will be managed exactly like everything else you operate, which matters when an auditor asks for change history. Choose Control Tower only if platform ownership is genuinely nobody's job yet and you are willing to live inside its rails as the price of AWS maintaining the baseline. Do not adopt AFT at this scale.

### Identity: one door, many rooms

IAM Identity Center from day one, federated to Google Workspace or Entra, with SCIM sync so offboarding cascades automatically. Zero per-account IAM users for humans. Access is expressed as permission sets assigned to groups per account: ReadOnly everywhere, Developer (write in staging, read in prod), PlatformAdmin, SecurityAudit. Session duration is a control, not a default: 8 hours for staging work, 1 hour for anything administrative in prod. The compliance dividend is real: a quarterly access review becomes an export of assignments per account instead of an archaeology dig through IAM users, and that export feeds your evidence pipeline directly.

Break-glass is the exception that proves the design. Two IAM users in the management account, hardware MFA, credentials sealed offline, and an EventBridge rule on their console logins that pages whoever is on call. Test the procedure twice a year and log that you did; an untested break-glass account is a locked fire exit.

### Networking: peer first, TGW when it earns it

Interrogate the mesh before building it. At this scale the honest connectivity requirement is usually shared-services to prod and shared-services to staging: two VPC peering connections, zero hourly cost. Transit Gateway runs about $36 per attachment per month plus $0.02/GB processed, and it earns that when you have four or more VPCs needing mutual reachability, a VPN or Direct Connect hub, or a centralized egress requirement. Centralized egress through the network account is genuinely attractive for regulated workloads (one choke point for flow logs and inspection) but it adds a data-processing toll and a shared failure domain, so let an assessor finding or an incident justify it rather than building it speculatively.

The one thing that is a day-one obligation: a non-overlapping CIDR plan, in IPAM or a spreadsheet, covering accounts you have not created yet. Nearly every other networking decision here is reversible. Overlapping CIDRs are not, and they surface at the worst moment, usually mid-migration.

### Landing the plane

Done is testable. All human access flows through Identity Center and the old IAM users are deleted. The org CloudTrail lands in log-archive, and an administrator attempting to permanently delete a log object version (a delete with `--version-id`) gets denied by Object Lock (run this test; the failure is the pass). An unversioned delete merely adds a delete marker, and the data remains recoverable underneath it. SCPs are attached and have been exercised from a workload account, not just written. The production workload actually runs in the prod account: an org where the accounts exist but the monolith still lives in the old account is scaffolding, not a landing zone. Budget alarms exist per account, and the org definition itself sits under [drift detection](/library/infrastructure-drift-detection).

Two connections worth making explicit. This topology is the grown-up version of the three-account split in [minimum viable HIPAA on AWS](/library/minimum-viable-hipaa-aws); if you built that, you retrofit into this by adding OUs and SCPs around accounts you already have. And the org trail flowing into log-archive is the raw feed for an [evidence pipeline](/library/evidence-pipelines-101), which is where this architecture starts paying for itself every audit cycle.

## The Template

The OU map, the two SCPs not shown above, and the Terraform repo layout.

```
Root
├── Management account        billing + org admin; runs nothing; root creds in a safe
├── Security OU
│   ├── log-archive           org CloudTrail + Config -> S3, Object Lock (compliance mode)
│   └── security-tooling      GuardDuty / Security Hub / Access Analyzer delegated admin
├── Infrastructure OU
│   ├── network               peering or TGW, IPAM, (later) central egress
│   └── shared-services       CI runners, ECR, artifact storage, internal tools
├── Workloads OU
│   ├── prod                  1h admin sessions, tightest SCPs
│   └── staging               mirrors prod topology; SCP canary target
└── Sandbox OU
    └── sandbox-<engineer>    $200 budget alarm, no peering, aws-nuke nightly
```

Region restriction, attached at the root (adjust the region list and the exempt global services to taste):

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyOutsideAllowedRegions",
      "Effect": "Deny",
      "NotAction": [
        "iam:*", "organizations:*", "sts:*", "cloudfront:*",
        "route53:*", "support:*", "budgets:*", "health:*"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:RequestedRegion": ["us-east-1", "us-west-2"]
        }
      }
    }
  ]
}
```

Log-archive protection, attached to the Security OU. The pipeline role is exempted; otherwise this SCP would also block the Terraform stack that manages the bucket's versioning, lock, and lifecycle configuration, and day-2 applies would fail:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ProtectLogArchiveBucket",
      "Effect": "Deny",
      "Action": [
        "s3:DeleteBucket",
        "s3:DeleteBucketPolicy",
        "s3:PutBucketVersioning",
        "s3:PutBucketObjectLockConfiguration",
        "s3:PutLifecycleConfiguration"
      ],
      "Resource": "arn:aws:s3:::org-log-archive-*",
      "Condition": {
        "ArnNotLike": {
          "aws:PrincipalArn": "arn:aws:iam::*:role/terraform-pipeline"
        }
      }
    }
  ]
}
```

Terraform repo layout, one state per stack, applied by a pipeline role:

```
terraform/
├── backend.hcl
├── org/                      # runs against the management account
│   ├── main.tf               # organization, OUs, member accounts
│   ├── scps.tf               # policy documents + OU attachments
│   └── identity.tf           # Identity Center permission sets + assignments
├── security/
│   ├── log-archive/          # bucket, Object Lock, org CloudTrail
│   └── tooling/              # GuardDuty / Security Hub delegated admin
├── infrastructure/
│   └── network/              # VPCs, peering or TGW, IPAM pools
└── modules/
    ├── organizations/        # OU + account factory (tags, close-on-delete)
    ├── scp/                  # policy rendering + attachment
    ├── log-archive/          # bucket with lock, replication, lifecycle
    ├── identity/             # permission set + group assignment pattern
    └── network/              # standard VPC shape, flow logs on by default
```

## Operating Notes

### The account count wants to grow. Let it, slowly.

A new account is justified by a new blast radius worth isolating: a new compliance boundary, a workload with its own on-call, a data classification that must not share IAM space with the rest. It is not justified by a new microservice or a new team lead's preference. Ten to fifteen accounts is the healthy range for this size of company. If you are at forty, you have rebuilt the governance problem you started with, except now it has a monthly baseline bill in every account — Config, GuardDuty, and Security Hub all meter per account — and nobody remembers what `prod-svc-legacy-2` does.

### SCPs are production changes

An SCP typo can lock every engineer out of every workload account simultaneously, which makes the SCP repo the highest-blast-radius code you own. Treat it accordingly: PR review with plan output, rollout through the staging OU with a soak period before it touches Workloads, and a CloudWatch alarm on `organizations:*` mutations from any principal that is not the pipeline role. The upside in a regulated shop is that this discipline is self-documenting. Your SCP git history, with reviews and timestamps, is change-control evidence an auditor accepts without a follow-up meeting.

### Retrofitting under a live workload

The landing zone is designed so production never has to move. Invite the existing account into the org and place it under the Workloads OU as prod, exactly where it belongs. Build the new accounts around it: log-archive first (start capturing the org trail immediately, since evidence accrues from day one), then Identity Center, then shared-services, and migrate CI runners and staging out of the old account at leisure. Apply SCPs to the staging OU, watch for a week, then extend to prod. The anti-pattern is the migration weekend: a planned big-bang move of production into a "clean" account, which converts a governance project into an availability incident. Prod stays put. The org grows around it.

## Frequently asked questions

**Q: What is an AWS landing zone?**

A landing zone is a multi-account AWS setup built on AWS Organizations: one account per isolation boundary, grouped into OUs and constrained by service control policies, with identity unified through IAM Identity Center. For a 10-80 engineer regulated SMB it's typically five OUs and about ten accounts. It's the foundational layer you build before workloads sprawl.

**Q: Why isn't one AWS account with tags good enough?**

The account is the only hard security wall AWS gives you, so tag-based separation means a leaked CI token can reach production. Quotas like Lambda concurrency and EC2 vCPU are account-scoped, so a load test in staging can take prod down with it. And a SOC 2 or HIPAA assessor asking how production is isolated won't accept 'a tag' as the answer.

**Q: What are service control policies and how should I use them?**

SCPs are a deny-list, not a grant — an action is allowed only if IAM permits it AND no SCP denies it. A good starter set denies leaving the org, blocks CloudTrail tampering, denies root-user actions, restricts regions to the ones you operate in, and protects the log-archive bucket. Note they never apply to the management account, which is why nothing should run there.

**Q: Should I use AWS Control Tower or build the landing zone with Terraform?**

If you already run Terraform through CI and one engineer owns platform, roll your own in plain Terraform and skip Control Tower — the org config is then managed like everything else you operate, which matters when an auditor asks for change history. Choose Control Tower only if platform ownership is genuinely nobody's job yet. Don't adopt AFT at this scale.

**Q: How do I move to a multi-account setup without a risky migration?**

You don't move production. Invite the existing account into the organization and place it under the Workloads OU as prod, then build the new accounts around it — log-archive first, then Identity Center, then shared-services. Apply SCPs to the staging OU, watch for a week, then extend to prod. The planned migration weekend is the anti-pattern.

**Q: How many AWS accounts should a startup have?**

Ten to fifteen is the healthy range for a 10-80 engineer company. Add an account for a new blast radius worth isolating — a new compliance boundary, a workload with its own on-call, a data classification that shouldn't share IAM space — not for every microservice. At forty accounts you've rebuilt the original governance problem, now with a per-account Config, GuardDuty, and Security Hub bill.

Template: "Landing zone reference kit" — branded PDF at https://kaansystems.com/templates/multi-account-aws-landing-zone.pdf

Source: https://kaansystems.com/library/multi-account-aws-landing-zone · Work with Kaan Systems: https://kaansystems.com/contact

---

# On-Call Without Burning Out the Team

Canonical: https://kaansystems.com/library/on-call-without-burnout
Topic: Engineering Practice · Published: 2026-07-29 · Updated: 2026-10-12
Summary: On-call is where reliability and retention collide. Done badly it burns out your best engineer and trains everyone to ignore alerts. Four levers — a real rotation, alert quality, a severity ladder, and recovery — make it sustainable.

## Key takeaways

- On-call at a small company fails when it's a person, not a system: one engineer is effectively always on the pager, which is both a reliability risk and a retention risk.
- A humane rotation needs at least three or four people who can hold the pager, with one-week shifts as the common default and a schedule published months ahead so people can plan their lives.
- A page that wakes someone should fire only when the problem is both user-facing and actionable; everything else belongs on a dashboard, a daily digest, or a business-hours channel.
- Watch pages per person per week: sustained above two or three, your alerts are too loud and the team is being trained to ignore the one that matters.
- Three severity levels is enough, and every rotation needs a named secondary who auto-escalates when the primary doesn't answer — escalating to the one person who's always on is the burnout you're trying to fix, not a policy.
- Recovery is what makes on-call sustainable: compensate off-hours work, keep postmortems blameless, and feed every incident back into alert tuning so the rotation gets quieter over time.

**How do you run on-call without burning out the team?** Make it a system, not a person. Four levers do it: a real rotation of three or four people, alerts that page only on user-facing and actionable problems, a three-level severity ladder with a secondary who answers, and recovery: comp time, blameless postmortems, and constant alert tuning.

Pair the rotation with [SLOs a small team can keep](/library/slos-for-small-teams) and a cluster that is [production-ready](/library/production-ready-kubernetes) enough to debug at 3 AM. The [7-day Assessment](/assessment) looks at both.

## Why does on-call burn out a small team?

On-call at a small company usually isn't a system. It's a person. One engineer — often the best one, the one who knows where the bodies are buried — is effectively always on. Alerts fire constantly, most of them noise, so everyone has learned to ignore them, which means the one that matters gets ignored too. There's no real rotation, no escalation path, and no recovery time. It works right up until that person takes a real vacation, or quits.

This is two problems wearing one coat. It's a **reliability** problem: an alert everyone ignores is worse than no alert, and a single point of human failure is a single point of failure. And it's a **retention** problem: nothing burns out a strong engineer faster than being permanently, silently responsible for production at 2 a.m.

The fix isn't heroics or a bigger tool. It's turning on-call into a system with four moving parts, each of which you can design deliberately.

## Which four levers make the pager livable?

Four levers. They reinforce each other: a real rotation only works if the alerts are worth waking up for, and good alerts only stay good if there's recovery time to keep tuning them.

```mermaid
flowchart LR
  R[Rotation<br/>Share the load<br/>fairly + predictably]
  A[Alert quality<br/>Page only on<br/>user-facing + actionable]
  E[Escalation<br/>Severity ladder<br/>+ a backup who answers]
  C[Recovery<br/>Comp time, no blame<br/>keep tuning]

  R --> A --> E --> C
  C -.feeds back.-> A

  classDef lever fill:#1F4A3E,stroke:#5BC2A5,stroke-width:1.5px,color:#E8F5EE;
  class R,A,E,C lever;
```

### A real rotation, not "everyone, always"

The first lever is the simplest and the most resisted: put on-call on a schedule, with named people and defined shifts, and take everyone else off.

- **Minimum viable rotation is three or four people.** Fewer than that and the rest between shifts is too short to matter. If you only have two or three engineers who *can* hold the pager, that's your real problem — spread the operational knowledge before you spread the rotation.
- **One-week shifts are the common default.** Long enough to see patterns and own the follow-up, short enough to end. Some teams do shorter primary shifts to protect sleep; either works if it's predictable.
- **Predictable beats fair-in-theory.** People can plan their lives around a schedule they can see three months out. They can't plan around "whoever's around."

The goal of the rotation is that when you're not on it, you're genuinely off it. That's the part that prevents burnout, and it's the part small teams skip.

### Alert quality: page only on user-facing and actionable

A rotation is only humane if being on-call is mostly uneventful. The lever that makes it uneventful is ruthless alert quality. The bar for a page — the kind that wakes someone — is two tests, both required:

- **User-facing.** Is a customer being hurt right now? If the symptom is "a background queue is 5% deeper than usual" and no user notices, that's a business-hours ticket, not a 2 a.m. page.
- **Actionable.** Is there something the responder can actually do right now? An alert with no runbook and no action is just anxiety with a sound.

Everything that fails either test gets demoted: to a dashboard, to a daily digest, to a business-hours Slack channel. The number that tells you whether you've got this right is **pages per person per week.** Above two or three sustained, your alerts are too loud and your team is being trained to ignore them. The work of on-call, between incidents, is tuning that number down.

### A severity ladder and an escalation that answers

When something does fire, the responder needs to know how hard to hit it, and needs a backup who will actually pick up.

- **Three severity levels is enough.** Sev1: customer-facing outage, page immediately, all hands if needed. Sev2: degraded but working, page during hours / next morning otherwise. Sev3: ticket. Write the definitions down so the 2 a.m. judgment is already made.
- **Every rotation has a secondary.** The primary doesn't answer within the SLO — laptop dead, dead asleep, in a tunnel — and it escalates automatically to a named secondary, then to a lead. "Escalate to the person who's always on" is not an escalation policy; it's the burnout you're trying to fix.
- **Escalating is not failure.** Make it explicit and cultural: waking up the secondary because you're stuck is the system working, not the responder failing.

### Recovery: comp time, no blame, keep tuning

The last lever is what makes the whole thing sustainable over months instead of collapsing back to the hero model.

- **Compensate real off-hours work.** If someone was up for two hours at 3 a.m., they start late, or take the time back. Unpaid, uncompensated night work is a debt you're taking out against retention.
- **Blameless everything.** The postmortem asks what in the system let this happen, never who screwed up. The moment on-call feels like exposure to blame, your best people stop volunteering to respond.
- **Feed incidents back into alert tuning.** Every page is data. A recurring low-value page gets demoted; a missing alert that should have fired gets added. The rotation that tunes itself gets quieter over time; the one that doesn't gets louder until people leave.

## The Template

An on-call policy your team can actually run. Fill it in, publish it where everyone can find it, and revisit it quarterly.

**Rotation**
- Responders in the rotation: _____________ (target: 4+)
- Shift length: _______ (default: 1 week)
- Schedule published _______ months ahead
- Handoff ritual: ________________________ (what the outgoing responder passes on)

**What pages (the bar)**
- A page requires BOTH: user-facing impact AND a clear action
- Everything else routes to: ☐ dashboard ☐ daily digest ☐ business-hours channel
- Target pages per person per week: ______ (alert if sustained above 2-3)

**Severity ladder**
- Sev1 — customer-facing outage: page immediately, escalate to all-hands if unresolved in ____ min
- Sev2 — degraded / at-risk: page in hours; next morning off-hours
- Sev3 — non-urgent: ticket, next business day

**Escalation**
- Primary responder SLO to acknowledge: ______ minutes
- Auto-escalate to SECONDARY after: ______ minutes unacknowledged
- Then escalate to: ______________ (lead / manager)
- Secondary is a named person every shift: ☐ yes

**Recovery**
- Off-hours response is compensated by: ☐ time back ☐ late start ☐ ______
- Postmortems are blameless: ☐ yes, documented
- Every incident reviewed for an alert-tuning action: ☐ yes

## Operating Notes

Watch two numbers, not one. Everyone tracks incident count. The number that predicts burnout is **off-hours pages per person** — the 2 a.m. wakeups, specifically. A team can absorb a lot of daytime pages and very few nighttime ones. If the nighttime number is climbing, fix it before someone quits, because by the time they quit it's too late to fix cheaply.

Rotate the boring work too. The person who always writes the runbooks and the person who always takes the ugly shift are accumulating invisible debt. Rotate runbook authorship, rotate the holiday coverage, rotate the secondary. On-call fairness is mostly about the parts nobody volunteers for.

If you can't staff a humane rotation yet, that's the signal — not that on-call is broken, but that too few people can safely hold production. The fix is spreading operational knowledge: pairing on incidents, writing the runbooks, doing the game-day drills. A rotation is the reward for having enough people who can respond. Building that bench is the actual work, and it's worth starting before the hero burns out.

## Frequently asked questions

**Q: How many people do you need for an on-call rotation?**

A humane rotation needs at least three or four people who can hold the pager — fewer than that and the rest between shifts is too short to matter. If only two or three engineers can safely respond, that's your real problem: spread the operational knowledge before you spread the rotation.

**Q: What should actually trigger a 2 a.m. page?**

A page that wakes someone has to pass two tests, both required: the problem is user-facing (a customer is being hurt right now) and actionable (there's something the responder can do immediately). Anything that fails either test gets demoted to a dashboard, a daily digest, or a business-hours channel.

**Q: How many pages per week is too many?**

Track pages per person per week. Sustained above two or three, your alerts are too loud and your team is being trained to ignore them. The work of on-call between incidents is tuning that number down.

**Q: How many severity levels does an on-call policy need?**

Three is enough. Sev1 is a customer-facing outage (page immediately, all hands if needed), Sev2 is degraded-but-working (page during hours, next morning otherwise), and Sev3 is a ticket. Write the definitions down so the 2 a.m. judgment is already made.

**Q: How do you keep on-call from burning people out?**

Make it a system with recovery built in: compensate real off-hours work with time back or a late start, keep every postmortem blameless, and feed incidents back into alert tuning. The number that predicts burnout is off-hours pages per person — the 2 a.m. wakeups specifically — so fix it before someone quits.

**Q: What if we don't have enough people to staff a rotation?**

That's the signal that too few people can safely hold production, not that on-call is broken. The fix is spreading operational knowledge — pairing on incidents, writing the runbooks, running game-day drills — so you build the bench before the hero burns out.

Template: "On-call rotation + escalation policy" — branded PDF at https://kaansystems.com/templates/on-call-without-burnout.pdf

Source: https://kaansystems.com/library/on-call-without-burnout · Work with Kaan Systems: https://kaansystems.com/contact

---

# Vendor Risk Scoring for Regulated SMBs

Canonical: https://kaansystems.com/library/vendor-risk-scoring
Topic: Security · Published: 2026-07-29 · Updated: 2026-10-12
Summary: A vendor breach becomes your breach. Most SMBs 'assess vendors' with a gut check at procurement and never again. A lightweight four-factor rubric produces a repeatable risk score, sorts vendors into tiers, and sets a review cadence that matches the risk.

## Key takeaways

- Third-party risk is a large share of a regulated SMB's attack surface, and it's the share zero-trust controls can't reach — you can't enroll a vendor in your MDM or put them behind your access proxy.
- Score each vendor 1-3 on four factors — data sensitivity, access and integration depth, attestation gap, and blast radius — then sum them for a 4-12 risk score.
- Data sensitivity is the biggest lever, so weight it heaviest: a vendor that can read patient records and one that resizes marketing images aren't the same risk, even with identical SOC 2 reports.
- The score maps to a tier — Critical 10-12, High 7-9, Medium 5-6, Low 4 — and the tier, not gut feel, decides how much review effort each vendor gets.
- Score every vendor at procurement before access is granted, and re-score immediately whenever a vendor discloses a breach, not just on the annual calendar.
- The rubric is triage, not certainty: it tells you which five vendors out of forty deserve deep review, turning 'review all our vendors' into 'review the five that matter'.

**How do you score vendor risk without a GRC platform?** Rate each vendor 1-3 on four factors — data sensitivity, access depth, attestation gap, and blast radius — and sum them into a 4-12 score. The score sorts vendors into tiers, and the tier sets the review cadence, so deep review goes to the five vendors that actually matter, not all forty.

## Why does a gut-feel vendor review fail?

Your security floor is your vendors' security floor. A vendor you gave access to sensitive data gets breached, and now you're breached — Snowflake credentials in 2024, MOVEit in 2023, and a long tail of quieter incidents that became someone else's disclosure notice. Third-party risk isn't a side concern for a regulated SMB. It's a large share of the actual attack surface — and the share the [zero-trust controls you apply to your own users and devices](/library/zero-trust-for-smbs) can't reach, because you can't enroll a vendor in your MDM or put them behind your access proxy.

The way most SMBs handle it: a gut-feel check at procurement ("seems legit, they have a SOC 2"), a signed order form, and then nothing, ever again. By the time the company has 40 SaaS vendors, nobody can say which ones can read customer data, which ones have an attestation, or which one would take the business down if it disappeared on a Tuesday.

You can't review 40 vendors deeply every year, and you shouldn't try. What you need is a cheap, repeatable way to **score** each vendor so the review effort goes where the risk is. Not a GRC platform. A rubric. Keep the attestation you accepted queryable in an [evidence pipeline](/library/evidence-pipelines-101), and reuse that proof when you [answer a security questionnaire](/library/reading-security-questionnaire). The [7-day Assessment](/assessment) is the read-only pass over the estate those vendors touch.

## How do you score a vendor on four factors?

Score each vendor on four factors. Multiply the sensitivity of the data by how deeply the vendor is wired in, adjust for how much assurance they've given you and how much a breach would cost — that's the risk. The score sorts vendors into tiers, and the tier sets the review cadence.

```mermaid
flowchart LR
  F1[Data sensitivity<br/>What can they touch?]
  F2[Access + depth<br/>How wired in are they?]
  F3[Attestation gap<br/>What have they proven?]
  F4[Blast radius<br/>What if they're breached?]

  F1 --> S[Risk score]
  F2 --> S
  F3 --> S
  F4 --> S
  S --> T{Tier}
  T --> C1[Critical / High<br/>review at procurement<br/>+ annually]
  T --> C2[Medium / Low<br/>review at procurement<br/>+ on change]

  classDef factor fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
  classDef score  fill:#1F4A3E,stroke:#3FA585,stroke-width:2px,color:#E8F5EE;
  classDef gate   fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
  classDef tier   fill:#1F4A3E,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
  class F1,F2,F3,F4 factor;
  class S score;
  class T gate;
  class C1,C2 tier;
```

### Factor 1: Data sensitivity

What class of data can this vendor touch? This is the biggest lever, so weight it heaviest.

- **3 — Regulated:** PHI, PII, cardholder data, or credentials to systems holding those.
- **2 — Confidential:** internal business data, customer lists, non-regulated but sensitive.
- **1 — Low / none:** public data, or a tool that never touches customer data at all.

A vendor that can read your patient records and a vendor that resizes your marketing images are not the same risk, no matter how similar their SOC 2 reports look.

### Factor 2: Access and integration depth

How wired in are they? A breach of a deeply-integrated vendor with standing API access is a very different event from a breach of a tool someone logs into occasionally.

- **3 — Deep / standing:** persistent API access, OAuth into your core systems, an agent on your infrastructure.
- **2 — Moderate:** regular data flow, scoped integration, SSO with meaningful permissions.
- **1 — Shallow:** occasional manual use, no standing access, easily revoked.

### Factor 3: Attestation gap

What have they actually proven about their security? This factor *reduces* risk when it's strong, so score the gap: high score means little assurance.

- **3 — Nothing:** no SOC 2, no ISO 27001, no security page, won't answer a questionnaire.
- **2 — Partial:** a trust page and some answers, or an attestation that's stale or narrow.
- **1 — Strong + current:** current SOC 2 Type II or equivalent, covering the relevant scope.

A current, in-scope attestation isn't proof of safety, but its absence is a real signal — especially from a vendor touching regulated data.

### Factor 4: Blast radius

If this vendor vanished or was compromised, what does it cost you? This captures operational dependence, not just data exposure.

- **3 — Business-critical:** losing them takes down your product or a core business function.
- **2 — Significant:** painful, workaround exists, degraded operations for days.
- **1 — Minor:** annoying, quickly replaced or lived without.

### From score to tier to cadence

Sum the four factors (4–12). The score sorts each vendor into a tier, and the tier — not your gut — decides how much review effort they get:

- **10–12 — Critical:** security review at procurement, annual re-review, notification requirement in the contract, on your sub-processor list if they touch customer data.
- **7–9 — High:** review at procurement, annual re-review, attestation on file.
- **5–6 — Medium:** review at procurement, re-review on material change (new integration, new data access).
- **4 — Low:** lightweight check at procurement, no scheduled re-review.

The point isn't the exact numbers. It's that a vendor with standing access to PHI and no SOC 2 scores an 11 and gets real scrutiny, while the image resizer scores a 4 and doesn't consume your limited review time.

## The Template

Score each vendor across the four factors. Multiply data sensitivity by nothing — just sum — but if you want the sensitivity to dominate, weight Factor 1 double. Keep it simple enough that you'll actually run it.

| Vendor | Data sensitivity (1-3) | Access depth (1-3) | Attestation gap (1-3) | Blast radius (1-3) | Score (4-12) | Tier | Next review |
|---|---|---|---|---|---|---|---|
| _Example: analytics vendor_ | 2 | 2 | 1 | 2 | 7 | High | + annual |
| _Example: PHI data processor_ | 3 | 3 | 2 | 3 | 11 | Critical | + annual |
| _Example: image CDN_ | 1 | 1 | 1 | 1 | 4 | Low | at change |
| | | | | | | | |
| | | | | | | | |

**Tier thresholds**
- 10-12 Critical — procurement review + annual + contract notification clause + sub-processor list
- 7-9 High — procurement review + annual + attestation on file
- 5-6 Medium — procurement review + re-review on material change
- 4 Low — lightweight check at procurement

**Scoring rules**
- Score at procurement, before access is granted, and record the score.
- Re-score any vendor immediately if they disclose a breach — a Critical vendor's incident is your incident.
- The sub-processor list on your trust page is the source of truth for who touches customer data; every vendor scoring on Factor 1 at 2+ belongs on it.

## Operating Notes

Score at the gate, not after. The cheapest time to assess a vendor is before you've integrated them, when "no" is still easy and you have leverage to ask for the DPA, the attestation, or the notification clause. A vendor scored after they're wired into production is a vendor you're stuck with. Make the score a required field on the procurement checklist.

Re-score on their breach, not just your calendar. The annual cadence is the floor. The real trigger for a re-score is a vendor disclosing an incident of their own — that's the moment their risk to you changed, and it's exactly when most SMBs do nothing because there's no process pointing at it. A Critical-tier vendor's breach notification should automatically open a review on your side.

The rubric's job is triage, not certainty. A four-factor score won't tell you a vendor is safe. It tells you which five vendors out of forty deserve the deep review your team actually has time for, and which thirty-five don't. That triage is the whole value: it turns "review all our vendors" — a task that never happens — into "review the five that matter," which does.

## Frequently asked questions

**Q: How do you score vendor risk for a small business?**

Rate each vendor 1 to 3 on four factors — data sensitivity, access and integration depth, attestation gap, and blast radius — then sum them into a 4-12 score. The score sorts each vendor into a tier, and the tier sets how often you review them. It's a rubric, not a GRC platform.

**Q: What are the four factors in the vendor risk rubric?**

Data sensitivity (what class of data the vendor can touch), access and integration depth (how wired in they are), attestation gap (how little they've proven about their security), and blast radius (what it costs you if they're compromised or vanish). Data sensitivity is the biggest lever, so weight it heaviest.

**Q: How often should I review my vendors?**

Let the tier decide. Critical (10-12) and High (7-9) vendors get a security review at procurement plus an annual re-review; Medium (5-6) vendors are re-reviewed on material change; Low (4) vendors get a lightweight check at procurement and no scheduled re-review.

**Q: When should I re-score a vendor?**

The annual cadence is the floor. Re-score immediately if a vendor discloses a breach of their own — that's the moment their risk to you changed, and a Critical-tier vendor's incident is your incident. Most SMBs do nothing here because no process points at it.

**Q: Does a vendor's SOC 2 report mean they're safe?**

No. A current, in-scope attestation isn't proof of safety, but its absence is a real signal — especially from a vendor touching regulated data. The rubric scores the attestation gap as one of four factors, not as a pass/fail verdict.

**Q: Do I need a GRC platform to do vendor risk scoring?**

No. The whole point is a cheap, repeatable rubric you'll actually run — four factors, a 4-12 score, and a tier that sets the review cadence. Its job is triage: telling you which five vendors out of forty deserve the deep review your team actually has time for.

Template: "Vendor risk scoring rubric" — branded PDF at https://kaansystems.com/templates/vendor-risk-scoring.pdf

Source: https://kaansystems.com/library/vendor-risk-scoring · Work with Kaan Systems: https://kaansystems.com/contact

---

# What Enterprise Buyers Actually Ask About Your AI Features

Canonical: https://kaansystems.com/library/what-enterprise-buyers-ask-ai
Topic: AI · Published: 2026-07-29 · Updated: 2026-10-12
Summary: As you add AI features, enterprise security questionnaires grow an AI section. Buyers aren't testing whether your AI is flawless — they're testing whether you've thought about the AI-specific risks. Five themes, four answer patterns, fifteen pre-filled responses.

## Key takeaways

- Adding AI features grows your enterprise security questionnaire by roughly twenty AI-specific questions, and they aren't testing whether your model is perfect — they're testing whether you've thought about the AI-specific risks well enough to answer honestly.
- The AI questions cluster into five themes: data handling, vendor and sub-processor, output safety and human oversight, isolation, and governance. Recognize the theme and you know what the buyer is actually worried about.
- The same four answer patterns from the core questionnaire apply — yes with evidence, yes with honest scope, compensating control, and not applicable; only the subject matter changes.
- Data handling is the theme buyers care about most, and an undisclosed AI sub-processor discovered mid-review is the fastest way to lose the deal — name the vendor, confirm the agreement, and point to your public sub-processor list.
- Build a versioned answer bank keyed by question, tag each answer with a 'last verified' date, and re-check anything older than three months, since the AI section changes faster than the rest of the questionnaire.
- You don't need an AI ethics board; you need to show a documented process exists. The teams that pass describe their unremarkable, well-governed AI in five honest sentences — that legibility is the product.

**What do enterprise buyers actually ask about your AI features?** The questions cluster into five themes — data handling, vendor and sub-processor, output safety, isolation, and governance. None test whether your model is perfect; they test whether you know where the data goes, who the vendor is, and what happens when the model is wrong. Answer that honestly and you pass.

## What do enterprise buyers ask about AI features?

You added AI features. Now the enterprise security questionnaire that used to have 240 questions has 260, and the new twenty are all about the AI. "Do you train on our data?" "Where do prompts go?" "Can the model take an action that can't be undone?" "Is our data isolated from other customers' at the model layer?"

Teams freeze on these, for the same reason they froze on the original questionnaire before they'd seen a few: the questions look like they're testing whether your AI is safe, and you're not sure it is. It reads like an exam you might fail.

It isn't. The AI section tests exactly what the rest of the questionnaire tests — see [reading an enterprise security questionnaire](/library/reading-security-questionnaire) for the core version. It's checking whether **you've thought about the AI-specific risks well enough to answer honestly.** A CISO doesn't expect your Series B startup to have a formal AI governance board. They expect you to know where the data goes, who the model vendor is, and what happens when the model is confidently wrong. Answer that, accurately, and you pass.

## How do you answer without claiming a perfect model?

The AI questions cluster into five themes. Recognize the theme and you know what the buyer is actually worried about, which tells you what a good answer looks like.

```mermaid
flowchart TD
  Q[An AI question on the questionnaire]
  Q --> T1[Data handling<br/>Do you train on our data?<br/>Where do prompts go?]
  Q --> T2[Vendor + sub-processor<br/>Which model vendor?<br/>BAA / DPA in place?]
  Q --> T3[Output safety<br/>Can the AI take<br/>irreversible action?]
  Q --> T4[Isolation<br/>Is our data isolated<br/>at the model layer?]
  Q --> T5[Governance<br/>Who approves AI features?<br/>Plan for a bad output?]

  classDef q     fill:#0d1f1a,stroke:#6FE3B5,stroke-width:1.5px,color:#E8F5EE;
  classDef theme fill:#1F4A3E,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
  class Q q;
  class T1,T2,T3,T4,T5 theme;
```

The same four answer patterns from the core questionnaire apply: **yes with evidence**, **yes with honest scope**, **compensating control**, and **not applicable**. What changes is the subject matter.

### Theme 1: Data handling

The buyer's real fear: their data trains someone else's model, or ends up somewhere they can't see. This is the theme they care about most.

The questions: Do you train on our data? Are prompts and outputs logged, and where? What's the retention? Can we opt out of any data use?

The honest answer is usually short and reassuring, if you've done the work from [adding AI features without breaking your compliance posture](/library/ai-features-compliance-posture): "We do not train on customer data. Prompts and responses are processed by our model vendor under a no-training agreement, logged only in scoped-shape form for debugging (no content), and retained for 30 days." If any of that isn't true, the questionnaire is telling you what to go fix.

### Theme 2: Vendor and sub-processor

The buyer's real fear: you've added an AI vendor to their data supply chain and didn't tell anyone.

The questions: Which model provider do you use? Is there a BAA or DPA with them? Is the provider on your sub-processor list? Do they train on inputs?

The answer that builds trust names the vendor, confirms the agreement, and points to the public list: "AI inference runs on [vendor] under our signed [BAA/DPA]; they are listed on our public sub-processor page and do not train on our inputs." An undisclosed AI sub-processor discovered mid-review is the fastest way to lose the deal.

### Theme 3: Output safety and human oversight

The buyer's real fear: the AI does something consequential and wrong, automatically, to their data or their customers.

The questions: Can the model take actions autonomously? Is there human review before a consequential output? How do you handle hallucinations?

The pattern here is the compensating-control answer: "The model drafts; it does not decide. Consequential outputs require human review before they take effect. We monitor for and log low-confidence outputs." If your feature genuinely does act autonomously, say so and describe the guardrails honestly — a vague "we follow best practices" gets you a follow-up call you don't want.

### Theme 4: Isolation

The buyer's real fear: another customer's prompts leak into their results, or vice versa, through the model.

The questions: Is our data isolated at the model layer? Do you use our data in prompts for other customers (few-shot examples, RAG)? Could our data appear in another tenant's output?

The answer turns on your architecture. If each request is stateless and scoped to one tenant, say it plainly: "Prompts are constructed per-request from a single tenant's data. We do not use one customer's data as context for another's requests." If you use a shared vector store or cross-tenant examples, that's a real design question to resolve before you answer.

### Theme 5: Governance

The buyer's real fear: you're shipping AI features with no process, so the next one will be the reckless one.

The questions: Who approves AI features? Do you have an AI use policy? What's your incident plan for a harmful output?

You don't need an AI ethics board. You need to show a process exists: "New AI features run a documented compliance pre-flight before launch (data flow, vendor, output safety). Harmful outputs are handled under our standard incident response." A named, followed [AI governance framework](/library/ai-governance-for-regulated-products) beats an aspirational policy every time.

## The Template

Fifteen of the most common AI-security questions with answer patterns. Adapt the specifics to your architecture; keep the structure. Vague answers get flagged; brief, specific, honest answers get you to the next stage.

| # | Question | Answer pattern |
|---|---|---|
| 1 | Do you train AI models on customer data? | No. Customer data is never used to train or fine-tune models. |
| 2 | Which AI/LLM provider(s) do you use? | [Vendor(s)], accessed via API under [BAA/DPA]. Listed on our sub-processor page. |
| 3 | Does your AI provider train on your inputs? | No — confirmed in writing under our enterprise agreement; opt-out verified. |
| 4 | Are prompts and responses logged? Where? | Shape only (no content) in app logs. Any retained content is in-boundary, encrypted, scoped. |
| 5 | What is the retention for AI prompt/response data? | [X days]. Set deliberately; not left at vendor default. |
| 6 | Is customer data isolated at the model layer? | Yes. Prompts are constructed per-request, scoped to one tenant. No cross-tenant context. |
| 7 | Can the AI take autonomous/irreversible actions? | No. The model drafts or suggests; consequential actions require human review. |
| 8 | How do you handle hallucinations / incorrect output? | Human-in-the-loop on consequential outputs; low-confidence outputs flagged + logged. |
| 9 | Is there a human-in-the-loop for [use case]? | [Yes — where + how] OR [Compensating: output is advisory only, not acted on automatically]. |
| 10 | Do you have an AI acceptable-use policy? | [Yes — summary] OR [Documented AI feature pre-flight applied before each launch]. |
| 11 | How are AI features reviewed before release? | Documented compliance pre-flight: data flow, vendor agreement, output safety, logging. |
| 12 | Can we opt out of AI features / processing? | [Yes — how] OR [AI features are [optional/tiered]; describe the control]. |
| 13 | Is generated content labeled as AI-generated? | Yes. Users can distinguish AI-generated content. |
| 14 | What is your incident process for a harmful AI output? | Handled under standard IR; severity assessed; affected customers notified per policy. |
| 15 | Do you use our data in a shared vector store / RAG index? | [No — per-tenant isolation] OR [Yes — describe tenant scoping + access controls honestly]. |

These fifteen cover the large majority of any AI section. The rest are environment-specific — your exact model, your exact data flows — and those become the focused work once the baseline is in place.

## Operating Notes

Build the AI answer bank the same way you built the security one: every answer you write goes into a versioned document, keyed by question, so the next questionnaire is a copy-paste plus a delta review. The AI section changes faster than the rest, so tag each answer with a "last verified" date and re-check anything older than three months.

The "we don't train on your data" answer will be re-asked, re-verified, and eventually audited. Make sure it stays true. The single fastest way to destroy the trust this whole exercise builds is to answer "no training" while a vendor default quietly flipped to "training unless you opt out." Re-read your model vendors' terms on a schedule, not when a buyer asks.

The teams that handle the AI section well aren't the ones with the most sophisticated AI. They're the ones who can describe their unremarkable, well-governed AI in five honest sentences. That legibility is the product.

## Frequently asked questions

**Q: What do enterprise buyers ask about AI features in a security questionnaire?**

The AI questions cluster into five themes: data handling (do you train on our data, where do prompts go), vendor and sub-processor (which model provider, is there a BAA or DPA), output safety and human oversight, data isolation at the model layer, and governance. Recognize the theme and you know what the buyer is actually worried about.

**Q: Do enterprise buyers expect my startup to have formal AI governance?**

No. A CISO doesn't expect a Series B startup to have a formal AI governance board — they expect you to know where the data goes, who the model vendor is, and what happens when the model is confidently wrong. You need to show a documented process exists, not an ethics board.

**Q: How should I answer 'do you train on our data'?**

If it's true, keep it short and reassuring: you don't train on customer data, prompts are processed by your model vendor under a no-training agreement, logged only in scoped-shape form for debugging with no content, and retained for a set window. Make sure it stays true — re-read your vendor's terms on a schedule, because a default that quietly flips to 'training unless you opt out' destroys the trust this whole exercise builds.

**Q: What if my AI feature actually takes autonomous actions?**

Say so and describe the guardrails honestly. The trust-building pattern is 'the model drafts; it does not decide,' with human review before consequential outputs take effect, but a vague 'we follow best practices' just earns you a follow-up call you don't want.

**Q: How do I answer the AI section faster next time?**

Build the AI answer bank the same way as the security one: every answer goes into a versioned document keyed by question, so the next questionnaire is a copy-paste plus a delta review. Tag each answer with a 'last verified' date and re-check anything older than three months, since the AI section changes faster than the rest.

**Q: Does having more sophisticated AI help me pass the review?**

No. The teams that handle the AI section well aren't the ones with the most sophisticated AI — they're the ones who can describe their unremarkable, well-governed AI in five honest sentences. That legibility is the product.

Template: "Pre-filled responses to 15 common AI-security questions" — branded PDF at https://kaansystems.com/templates/what-enterprise-buyers-ask-ai.pdf

Source: https://kaansystems.com/library/what-enterprise-buyers-ask-ai · Work with Kaan Systems: https://kaansystems.com/contact

---

# SOC 2 Readiness in 90 Days: What Engineering Builds

Canonical: https://kaansystems.com/library/90-day-soc2-readiness-sprint
Topic: Compliance · Published: 2026-07-09 · Updated: 2026-10-12
Summary: Choosing SOC 2 is the easy part. The stall is having no plan. This is the quarter of engineering work: scope, policy, controls, and an evidence pipeline, in that order.

## Key takeaways

- SOC 2 readiness is roughly 90 focused days of building — scope, policy, controls, evidence, in that order — not the year-long project vendors imply; the multi-month observation window that follows is mostly waiting and collecting.
- Scope first, because scope determines everything downstream. Implementing controls before scoping means redoing work when the scope turns out smaller than feared.
- Only the Security criterion is mandatory. Add Availability if you make uptime commitments and Confidentiality if you handle sensitive data; skip Processing Integrity and Privacy unless a specific buyer asks.
- Write policies that describe what your team actually does. An aspirational policy like 'access is reviewed quarterly' when you've never run a review is a finding waiting to happen.
- Implement the control first, then let a compliance platform watch it — the tool automates evidence collection, it does not implement the control.
- Budget a dedicated owner. Ninety days with a designated owner stays ninety days. Squeezed into everyone's spare time, the same work becomes the year-long stall.

**How do you get SOC 2 ready in 90 days?** Run four sequenced phases: scope and gap-assess (weeks 1-2), write policies and stand up processes (weeks 3-5), implement the technical controls (weeks 6-9), then build the evidence pipeline and start your Type II observation window (weeks 10-12). Do them in order, with a dedicated owner.

## Why does SOC 2 readiness stall before it starts?

Once you've decided SOC 2 is the framework you need — see [choosing between SOC 2, HITRUST, and HIPAA](/library/choosing-soc2-hitrust-hipaa) if you haven't — the next question is deceptively hard: what's the actual plan?

[SOC 2](/library/soc2-for-startups-complete-guide) feels enormous and vague. There's no single checklist, the criteria are written for auditors, and every vendor selling you a tool has an incentive to make it sound like a year-long transformation. So teams do the thing that guarantees failure: they don't start. The project sits on the roadmap for three quarters while deals get lost on "you don't have SOC 2."

The work isn't a year. The work is roughly 90 focused days to get *ready* — to the point where a Type II observation window can start. The observation window itself then runs its course (three to twelve months of the controls operating), but that part is mostly waiting and collecting, not building. The 90 days is the build.

## What does engineering actually build, week by week?

Four phases, sequenced so each one unblocks the next. Do them in order. The most common way this goes wrong is teams implementing controls before they've scoped, then re-doing work when the scope turns out to be smaller than they feared.

```mermaid
flowchart LR
  P1["<b>Weeks 1-2</b><br/>Scope + gap<br/><br/>Define the system<br/>Pick trust criteria<br/>Honest gap assessment"]
  P2["<b>Weeks 3-5</b><br/>Policy + process<br/><br/>Write the policies<br/>Stand up the processes<br/>Assign the owners"]
  P3["<b>Weeks 6-9</b><br/>Implement controls<br/><br/>Close the technical gaps<br/>Access reviews, logging<br/>Change management"]
  P4["<b>Weeks 10-12</b><br/>Evidence + kickoff<br/><br/>Evidence pipeline live<br/>Readiness assessment<br/>Start the window"]
  DONE(["Type II observation<br/>window begins"])

  P1 --> P2 --> P3 --> P4 --> DONE

  classDef phase fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE
  classDef done  fill:#1F4A3E,stroke:#3FA585,stroke-width:2.5px,color:#E8F5EE
  class P1,P2,P3,P4 phase
  class DONE done
```

### Weeks 1-2: Scope and gap assessment

Scope first, because scope determines everything downstream. SOC 2 covers "the system" — you decide what that system is. Narrow it honestly to the production environment and the people and processes that touch customer data. A tighter scope is fewer controls, fewer policies, and less evidence, without weakening the report for the buyers who'll read it.

Pick your Trust Services Criteria. Security is mandatory. Add Availability if you make uptime commitments, Confidentiality if you handle sensitive customer data beyond the obvious. Skip Processing Integrity and Privacy unless a specific buyer is asking — each one you add is real additional work.

Then run an honest gap assessment against the criteria. Not "what do we wish we had" — what actually exists today, in writing, with evidence. Most teams discover the gaps cluster: the controls exist informally but aren't documented, or the logging is on but nobody reviews it. Name every gap. That list is the engineering backlog.

### Weeks 3-5: Policy and process

Auditors assess your controls against your documented policies. No policy, no control, regardless of what you actually do. This phase writes the documents and stands up the processes they describe.

The core policy set is smaller than it looks: information security, access control, change management, incident response, vendor management, risk assessment, business continuity, and a few supporting ones. Don't write them from scratch and don't copy a template you won't follow — write the version that describes what your team will actually do, then do that.

The trap here is writing aspirational policies. A policy that says "access is reviewed quarterly" when you've never done a review is a finding waiting to happen. Write what's true, or make it true this week.

### Weeks 6-9: Implement controls

Now close the technical gaps from Week 1. This is the longest phase because it's the real work. The controls that show up in almost every SOC 2:

- MFA and SSO across every system that touches customer data
- Quarterly access reviews, with the review actually performed and recorded
- Centralized logging with retention, and alerting that routes to a human
- Change management: every production change tied to a reviewed PR or ticket
- Encryption at rest and in transit, verified not assumed
- Vendor management: the sub-processor list and annual reviews
- Onboarding/offboarding with a documented, followed checklist

The teams that finish this phase on time are the ones that resisted buying a compliance platform first. The tool automates evidence collection; it does not implement the control. Implement the control, then let the tool watch it.

### Weeks 10-12: Evidence and kickoff

The observation window measures whether your controls *operated* over time, so you need the evidence to generate itself continuously. Stand up the [evidence pipeline](/library/evidence-pipelines-101) now, before the window starts, so month one of the window is already producing clean artifacts.

Run a readiness assessment — internal, or with your eventual auditor's readiness service. It surfaces the gaps you'll otherwise discover three months into the window, when they're expensive to fix. Close them.

Then start the window. From here it's operate-and-collect, not build. The build is done.

## The Template

The week-by-week checklist. Each item has a clear done state. Run it in order; don't start a phase until the prior one's blocking items are closed.

**Weeks 1-2: Scope + gap assessment**
- [ ] System boundary defined in writing (what's in scope, what's explicitly out)
- [ ] Trust Services Criteria selected (Security + only what buyers require)
- [ ] Gap assessment complete: every gap named, owner assigned
- [ ] Report type decided (Type I as a stepping stone, or straight to Type II)

**Weeks 3-5: Policy + process**
- [ ] Core policy set drafted and approved (8-10 policies)
- [ ] Each policy describes what the team will actually do (no aspirational controls)
- [ ] Process owners assigned for every recurring control
- [ ] Policies published where employees can find them, acknowledgement tracked

**Weeks 6-9: Implement controls**
- [ ] MFA + SSO enforced across in-scope systems
- [ ] First access review performed and recorded
- [ ] Centralized logging + alerting live, routes to on-call
- [ ] Change management enforced: prod changes tied to reviewed PRs
- [ ] Encryption at rest + in transit verified across in-scope stores
- [ ] Vendor / sub-processor list built, annual review scheduled
- [ ] Onboarding + offboarding checklists documented and in use

**Weeks 10-12: Evidence + kickoff**
- [ ] Evidence pipeline generating artifacts automatically
- [ ] Readiness assessment complete, findings closed
- [ ] Auditor engaged, observation window start date set
- [ ] Observation window started

## Operating Notes

The window is not a break. The single most common way teams fail their first Type II is treating the observation window as "done" and letting the controls lapse. The access review you performed in Week 6 has to happen again next quarter, on time, recorded. The evidence pipeline has to keep running. Compliance is a continuous practice, and the window is where that practice gets measured.

Budget for a dedicated owner. Ninety days with a designated owner stays ninety days. The same work squeezed into the margins of everyone's real job is the year-long stall this article opened with. The work isn't hard. It's just work, and work needs an owner.

The second framework is cheaper than the first. Almost everything you build here — the policies, the evidence pipeline, the access review cadence — carries directly into HITRUST, ISO 27001, or a HIPAA assessment later. The 90 days buys you SOC 2 and most of the readiness for whatever comes after it.

## Frequently asked questions

**Q: How long does it take to get SOC 2 ready?**

About 90 focused days to reach readiness — the point where a Type II observation window can start. The window itself then runs three to twelve months, but that part is mostly operating and collecting evidence, not building.

**Q: What are the phases of 90 days of SOC 2 engineering?**

Four, in order: scope and gap assessment (weeks 1-2), policy and process (weeks 3-5), implement controls (weeks 6-9), and evidence plus kickoff (weeks 10-12). Each phase unblocks the next, so don't start one until the prior phase's blocking items are closed.

**Q: Which Trust Services Criteria should I scope for SOC 2?**

Security is mandatory. Add Availability if you make uptime commitments and Confidentiality if you handle sensitive customer data beyond the obvious. Skip Processing Integrity and Privacy unless a specific buyer is asking, because each criterion you add is real additional work.

**Q: Do I need a compliance platform to get SOC 2?**

A platform automates evidence collection but doesn't implement the controls. The teams that finish the controls phase on time are the ones that resisted buying a platform first — implement the control, then let the tool watch it.

**Q: What happens after the 90 days of engineering?**

The Type II observation window begins, and from there it's operate-and-collect, not build. The most common way teams fail their first Type II is treating the window as done and letting controls lapse — the access reviews and evidence pipeline have to keep running on cadence.

**Q: Why do SOC 2 projects stall for a year?**

Because the project has no shape and no owner. Squeezed into the margins of everyone's real job, the 90 days of work becomes the year-long stall; a designated compliance owner is what keeps it to 90 days.

Template: "Week-by-week SOC 2 readiness plan" — branded PDF at https://kaansystems.com/templates/90-day-soc2-readiness-sprint.pdf

Source: https://kaansystems.com/library/90-day-soc2-readiness-sprint · Work with Kaan Systems: https://kaansystems.com/contact

---

# Adding AI Features Without Breaking Your Compliance Posture

Canonical: https://kaansystems.com/library/ai-features-compliance-posture
Topic: AI · Published: 2026-07-09 · Updated: 2026-10-12
Summary: Every healthcare-adjacent SaaS is racing to add AI features. Most haven't traced what that does to their HIPAA or SOC 2 posture. Four leak points, one pre-flight checklist, no surprises at the next audit.

## Key takeaways

- An LLM feature opens four new PHI leak points that never show up in a demo: the prompt (the context sent to the vendor), the vendor (a new sub-processor), the logs (prompts and responses stored for debugging and evals), and the output (a confidently wrong answer is a patient-safety event).
- If PHI goes into a prompt, the model vendor is processing PHI on your behalf and HIPAA requires a signed BAA — the major providers offer one only on request and not on every tier, and the free ChatGPT product doesn't qualify.
- If you can't get a BAA, there are two honest options: de-identify the data before it leaves your systems (Safe Harbor or Expert Determination), or don't build the feature that way.
- Prompt and response logs are the leak point teams miss most, because app logs, observability traces, and eval pipelines quietly become a PHI store that was never in your HIPAA scope — log the shape (token counts, latency, model version), not the content.
- The more consequential the output, the more a human has to be in the decision itself, not just notified of it — draft, don't decide.
- Run the pre-flight per feature and per data flow, not per company, and re-read every model vendor's data-use terms quarterly, since a default can flip from 'no training' to 'training unless you opt out' without any notice.

## Where does an LLM feature leak regulated data?

Every healthcare-adjacent SaaS is adding AI features right now. Summarize the chart note. Draft the prior-auth letter. Answer the patient's question. The demos are compelling and the pressure to ship is real.

Adding an LLM to a regulated product opens four new PHI leak points: the prompt leaving your systems, the model vendor acting as an unsigned sub-processor, prompt-and-response logs becoming an unplanned PHI store, and confidently wrong output. Run a four-gate pre-flight — BAA, sub-processor disclosure, PHI-grade logging, human review — before every feature ships.

The compliance implications almost never get traced before the feature ships. An LLM feature isn't a normal feature. It opens four new ways for protected health information to leave the boundary you spent a year building:

- **The prompt.** Whatever context you assemble to send to the model — the chart, the record, the message — is now leaving your systems for the vendor's.
- **The vendor.** The model API is a new sub-processor, processing PHI on your behalf, usually without a signed agreement.
- **The logs.** Prompts and responses get logged for debugging, evals, and observability. Those logs are now a PHI store you didn't architect.
- **The output.** A hallucinated dosage or a confidently wrong eligibility answer is a patient-safety event, not a bug.

None of these show up in a demo. All of them show up in an audit, a customer security review, or a breach investigation. The fix is to run the same pre-flight before every AI feature, the way you'd run a threat model before a new system. This pre-flight is one control inside a broader [AI governance framework](/library/ai-governance-for-regulated-products) — the standing discipline that keeps AI features inside the compliance boundary you already built.

## What do you check before the feature ships?

Treat every AI feature as a compliance change, not just a product change. Four gates, answered in order. If any gate fails, the feature doesn't ship until it's fixed.

```mermaid
flowchart TD
  START[AI feature proposed] --> Q1{Does it touch<br/>regulated data?}
  Q1 -- no --> SHIP[Standard product review]
  Q1 -- yes --> Q2{BAA signed with<br/>the model vendor?}
  Q2 -- no --> B1[De-identify, or stop.<br/>No PHI without a BAA.]
  Q2 -- yes --> Q3{Prompts + responses<br/>handled as PHI?}
  Q3 -- no --> B2[Fix logging + retention first]
  Q3 -- yes --> Q4{Human review on<br/>consequential output?}
  Q4 -- no --> B3[Add a human gate]
  Q4 -- yes --> GO[Ship with monitoring]

  classDef q    fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
  classDef good fill:#1F4A3E,stroke:#3FA585,stroke-width:2px,color:#E8F5EE;
  classDef stop fill:#0a1612,stroke:#B5503A,stroke-width:1.5px,color:#E8F5EE;
  class Q1,Q2,Q3,Q4 q;
  class SHIP,GO good;
  class B1,B2,B3 stop;
```

### Sign a BAA with the model vendor, or don't send PHI

This is the gate that stops most teams cold, because it's usually skipped. If protected health information is going into a prompt, the model vendor is processing PHI on your behalf. Under HIPAA that requires a Business Associate Agreement, the same as it does for AWS or your database host.

The major providers offer BAAs, but not on every tier and not by default. OpenAI signs one for API customers who request it and meet the terms; the free ChatGPT product does not qualify. Anthropic, Google, and AWS Bedrock have their own paths. The point is that the BAA is a specific artifact you have to go get, not something that comes with the API key.

If you can't get a BAA, or the feature routes through a vendor who won't sign one, you have two honest options: de-identify the data before it leaves your systems (Safe Harbor or Expert Determination), or don't build the feature that way. "We'll get the BAA later" is how PHI ends up in a training corpus.

### Treat the model API as a sub-processor

Once the BAA is signed, the vendor belongs on your sub-processor list — the same public list your customers review, the same list your DPA references. Add it. A customer security review that finds an undisclosed AI sub-processor processing their data is a trust problem you don't recover from quickly.

While you're there, read the data-use terms. The question that matters: **does the vendor train on your inputs by default, and can you turn it off?** For most enterprise API tiers the answer is "no training, opt-out by default," but the terms change and the default varies by product. Verify it in writing, and re-verify it when the vendor updates their terms.

### Prompts and responses are a new PHI store

This is the leak point teams miss most often, because it's invisible. Your app logs the prompt for debugging. Your observability stack captures the request and response in a trace. Your eval pipeline saves inputs and outputs to measure quality. Every one of those is now storing PHI, usually in a system that was never in your HIPAA scope.

The controls are the same ones from [securing PHI in observability data](/library/securing-phi-observability), applied to a new source:

- Don't log full prompts or responses for PHI-bearing features. Log the shape — token counts, latency, model version, a request ID — not the content.
- If you must retain prompts for evals, retain them in a scoped, encrypted store inside your HIPAA boundary, with the same access controls as your production database.
- Set the retention deliberately. A prompt log that keeps PHI for two years "just in case" is a two-year breach window.

### Put a human in the loop for anything consequential

An LLM will produce a confident, fluent, wrong answer. In a consumer app that's an annoyance. In a clinical or eligibility or financial context it's a safety event and a liability.

The rule that holds up: **the more consequential the output, the more the human has to be in the decision, not just notified of it.** Summarizing a note for a clinician to review — the clinician is the gate, fine. Auto-sending a prior-auth denial with no human review — not fine. Draft, don't decide. Surface, don't act. The human gate is also your best defense against the hallucination you didn't catch in testing.

## The Template

Run this before every AI feature that could touch regulated data. Each item is yes / no / not-applicable. Any `no` on a required item blocks the ship until it's resolved.

**Legal + vendor**
- [ ] BAA signed with the model vendor (or PHI is de-identified before it leaves our systems)
- [ ] Vendor added to the public sub-processor list
- [ ] Vendor data-use terms reviewed: confirmed no training on our inputs
- [ ] Vendor's own retention of our data is documented and acceptable

**Data flow**
- [ ] Traced every field that goes into the prompt; confirmed the minimum necessary
- [ ] Confirmed no PHI in URL parameters, feature flags, or analytics events for this feature
- [ ] Response path checked: PHI in the output isn't written anywhere out of scope

**Logging + retention**
- [ ] Prompts + responses are NOT logged in plaintext to general application logs
- [ ] Any retained prompts/responses live in a scoped, encrypted, in-boundary store
- [ ] Retention period for prompt/response data is set deliberately, not left at default
- [ ] Observability traces for this feature strip or exclude prompt/response content

**Output safety**
- [ ] Consequential outputs have a human in the decision, not just a notification
- [ ] Users can tell an AI generated the content
- [ ] There's a documented plan for when the model is confidently wrong

**Ongoing**
- [ ] This feature is in scope for the next access review + evidence pipeline
- [ ] A re-review triggers when the vendor changes terms or we change the data flow

The checklist takes an hour to run for a new feature. The alternative — finding an undisclosed sub-processor and an out-of-scope PHI log during a customer's security review — takes a lot longer and costs the deal.

## Operating Notes

Run the pre-flight per feature, not per company. The BAA you signed for the summarization feature doesn't cover the new vendor you're using for embeddings. Each distinct model vendor and each distinct data flow gets its own pass.

Watch the vendor terms. AI providers update their data-use and retention terms far more often than your database host does. Put a quarterly reminder on the calendar to re-read the terms for every model vendor on your sub-processor list. A default that flips from "no training" to "training unless you opt out" is the kind of change that doesn't send you an email.

The teams that get this wrong aren't reckless. They're moving fast on a feature that looks like every other feature and isn't. The pre-flight is cheap precisely because it turns an invisible compliance change into a visible, answerable checklist — before the feature ships, not after the audit finds it.

## Frequently asked questions

**Q: Do I need a BAA to use an LLM with PHI?**

Yes. If protected health information goes into a prompt, the model vendor is processing PHI on your behalf, and under HIPAA that requires a Business Associate Agreement — the same as it does for AWS or your database host. The major providers offer BAAs, but only on request and not on every tier; the free ChatGPT product doesn't qualify.

**Q: What if I can't get a BAA from the model vendor?**

You have two honest options: de-identify the data before it leaves your systems (Safe Harbor or Expert Determination), or don't build the feature that way. 'We'll get the BAA later' is how PHI ends up in a training corpus.

**Q: Where does PHI leak when you add an AI feature?**

Four places: the prompt (the context you assemble and send to the vendor), the vendor itself (a new sub-processor), the logs (prompts and responses captured for debugging, evals, and observability), and the output (a hallucinated dosage or wrong eligibility answer). None show up in a demo; all show up in an audit, a security review, or a breach investigation.

**Q: How should we log prompts and responses for a PHI-bearing feature?**

Don't log full prompts or responses to general application logs — log the shape (token counts, latency, model version, a request ID), not the content. If you must retain prompts for evals, keep them in a scoped, encrypted store inside your HIPAA boundary with a deliberately set retention period, not a default.

**Q: Does AI output need a human reviewer?**

For anything consequential, yes — the more consequential the output, the more the human has to be in the decision, not just notified of it. Summarizing a note for a clinician to review is fine; auto-sending a prior-auth denial with no human review is not. Draft, don't decide.

**Q: How often should we re-check the model vendor's terms?**

Quarterly. AI providers update their data-use and retention terms far more often than a database host does, and a default that flips from 'no training' to 'training unless you opt out' won't send you an email. Run the pre-flight per feature and per data flow too — a BAA for one vendor doesn't cover the next.

Template: "AI-feature compliance pre-flight checklist" — branded PDF at https://kaansystems.com/templates/ai-features-compliance-posture.pdf

Source: https://kaansystems.com/library/ai-features-compliance-posture · Work with Kaan Systems: https://kaansystems.com/contact

---

# Zero-Trust for Regulated SMBs Without the Enterprise Price Tag

Canonical: https://kaansystems.com/library/zero-trust-for-smbs
Topic: Security · Published: 2026-07-09 · Updated: 2026-10-12
Summary: Zero-trust is marketed as an enterprise platform you buy. The principle — never trust, always verify — is achievable for a regulated SMB with identity tools you likely already have. Four phases, sequenced, with an honest 'good enough' bar at each.

## Key takeaways

- Zero-trust is a principle — never trust, always verify — not a product; a regulated SMB can reach most of its benefit with identity tools it already pays for rather than a six-figure platform.
- Sequence is the whole discipline: run the four phases in order — identity, then device, then network, then continuous verification — because each phase makes the next one meaningful.
- Phase 1 (identity) is most of the value: SSO fronting ~90% of your SaaS, MFA on 100% of accounts, and phishing-resistant FIDO2/passkeys on every admin account, with named accounts replacing shared logins.
- Phase 2 (device) means every device touching regulated data is encrypted, updated, and enrolled in a lightweight MDM — you don't need a $50/seat EDR suite to clear that bar.
- Phase 3 (network) retires the VPN-as-moat: segment the regulated environment off the general network and replace 'connect and you're trusted' with per-application access that checks identity and device every session.
- The real bar isn't enterprise zero-trust; it's that a phished credential on an unmanaged device can't reach the PHI store — reachable this quarter with tools you mostly already own.

**Can a regulated SMB reach zero-trust without a six-figure platform?** Yes. Zero-trust is a principle — never trust, always verify — not a product. A 30-person SMB can reach roughly 80% of the benefit with identity tools it already owns, run in four sequenced phases: identity, device, network, then continuous verification.

## Why does a VPN moat fail?

"Zero-trust" has been thoroughly captured by enterprise sales. Say the phrase and a vendor appears to sell you a six-figure platform: a full ZTNA suite, an identity-governance product, a microsegmentation fabric, a year of professional services. For a Fortune 500 that's the right conversation. For a 30-person regulated SMB it's a reason to conclude zero-trust is out of reach and keep running the flat network behind a VPN.

That conclusion is wrong, and the flat-network-behind-a-VPN posture it justifies is the exact thing that gets SMBs breached. The VPN is a moat. The moat model assumes the attacker is outside. But the attacker's most reliable path in is a phished employee credential — [the threat that tops any realistic SMB threat model](/library/pragmatic-threat-model) — and once that credential is inside the moat, a flat network hands them everything.

Zero-trust is a principle, not a product: **never trust, always verify — verify every request as though the network is already compromised, because it is.** The principle is achievable for an SMB with tools you likely already pay for. The trick is sequencing, not spending.

## What order gets a team to zero trust?

Four phases, in order. The order matters more than the tooling — each phase makes the next one meaningful. Buying the microsegmentation product before you've consolidated identity is spending money on phase three while phase one is still open.

```mermaid
flowchart LR
  P1[Phase 1<br/>Identity<br/><br/>SSO + universal MFA<br/>One identity to revoke]
  P2[Phase 2<br/>Device<br/><br/>Managed, encrypted<br/>Known devices only]
  P3[Phase 3<br/>Network<br/><br/>Segment the flat network<br/>Kill VPN-as-moat]
  P4[Phase 4<br/>Verify<br/><br/>Conditional access<br/>Continuous logging]

  P1 --> P2 --> P3 --> P4

  classDef done fill:#1F4A3E,stroke:#3FA585,stroke-width:2px,color:#E8F5EE
  classDef next fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE
  class P1 done
  class P2,P3,P4 next
```

### Phase 1: Consolidate identity

This is the whole game, and most of the value. Zero-trust verifies *identity* on every request, so you need one authoritative identity per person and one place to revoke it.

- **SSO across every SaaS tool that supports it.** One identity provider — you likely already have Google Workspace or Microsoft Entra — fronting every app. The goal: offboarding a person is one action, not forty.
- **MFA universal, no exceptions.** Every account, every human. Phishing-resistant MFA (passkeys, FIDO2 hardware keys) for anyone with administrative access, because that's the credential worth stealing.
- **Kill the shared logins.** The `admin@` account that four people know the password to is the opposite of verifiable identity. Named accounts only. The same logic extends to machine credentials: [the stored API keys and IAM users that outlive the people who created them](/library/killing-long-lived-credentials) are shared secrets by another name.

Good-enough bar for an SMB: SSO fronting 90% of your SaaS, MFA on 100% of accounts, FIDO2 on every admin. You can reach this with tools you already own. No new platform required.

### Phase 2: Establish device trust

Verifying the person isn't enough if they're on a compromised machine. Phase two is knowing the device is one you manage.

- **Managed, encrypted devices for anyone touching regulated data.** Full-disk encryption enforced, OS current. A lightweight MDM (there are SMB-priced options, and Apple Business Manager plus a basic MDM covers a Mac fleet cheaply) gives you enforcement without an enterprise endpoint suite.
- **Known devices only for sensitive systems.** Access to the production console or the PHI store comes from a managed device, not from someone's personal laptop at a coffee shop.

Good-enough bar: every device that touches regulated data is encrypted, updated, and enrolled. You don't need a $50/seat EDR platform to clear this bar; you need enforcement of the basics.

### Phase 3: Segment the network

Now retire the moat. A flat network where a single compromised device can reach everything is the failure mode zero-trust exists to prevent.

- **Segment by sensitivity.** The production PHI environment is not on the same flat network as the office printer and the marketing laptops. Cloud-native segmentation (VPC boundaries, security groups) does most of this for free if you're already in AWS or GCP.
- **Retire the VPN-as-moat.** Replace "connect to the VPN and you're trusted" with [per-application access that checks identity and device every time](/library/affordable-ztna-growing-teams). Cloudflare Access, Tailscale, and Google's BeyondCorp-style access are SMB-affordable and often cheaper than the VPN concentrator they replace.

Good-enough bar: the regulated environment is isolated from the general network, and reaching it verifies identity and device per session rather than per VPN connection.

### Phase 4: Verify continuously

The first three phases are structural. Phase four keeps them honest over time.

- **Conditional access.** Access decisions consider identity, device posture, and context — a login from a new country on an unmanaged device gets challenged or blocked, not waved through.
- **Log and review.** Access events flow to the same centralized logging you built for compliance. The point of "always verify" is lost if nobody ever looks at what was verified.
- **Extend the principle to vendors.** Zero-trust doesn't stop at your own users and devices. Every third party with standing access to your systems is trust you've extended, so [score and re-review it on a cadence](/library/vendor-risk-scoring) rather than granting it once and forgetting.

Good-enough bar: anomalous access gets challenged automatically, and access logs are reviewed on the same cadence as your other security controls.

## The Template

The maturity model. Score each phase honestly — Not started / Partial / Done. Work top to bottom; a later phase built on an incomplete earlier one is wasted money.

**Phase 1 — Identity**
- [ ] SSO fronts every SaaS tool that supports it
- [ ] MFA enforced on 100% of human accounts, no exceptions
- [ ] Phishing-resistant MFA (FIDO2 / passkeys) on every admin account
- [ ] Shared / role logins eliminated; named accounts only
- [ ] Offboarding a person is a single revocation action

**Phase 2 — Device**
- [ ] Full-disk encryption enforced on every work device
- [ ] OS + critical updates enforced (not left to the user)
- [ ] Devices touching regulated data are enrolled in MDM
- [ ] Sensitive systems reachable only from managed devices

**Phase 3 — Network**
- [ ] Regulated environment segmented off the general/office network
- [ ] Cloud segmentation (VPC / security groups) enforces the boundary
- [ ] VPN-as-moat replaced with per-application access checks
- [ ] No single device can reach everything on a flat network

**Phase 4 — Continuous verification**
- [ ] Conditional access considers device posture + context, not just password
- [ ] Anomalous access (new device, new geo) is challenged or blocked
- [ ] Access events flow to centralized logging
- [ ] Access logs reviewed on a defined cadence

Most SMBs score Phase 1 at Partial and everything below at Not Started. That's a normal starting point. The path from there is one phase at a time, in order, using tools you mostly already own.

## Operating Notes

Sequence is the whole discipline. The reason zero-trust projects fail isn't the tooling — it's buying phase three before finishing phase one, then wondering why the expensive segmentation product didn't stop the phished credential that still had flat access to everything. Identity first, always.

Resist the platform pitch until you've earned it. The four phases above get an SMB to a genuine zero-trust posture using an identity provider you already have, a modest MDM, cloud-native segmentation, and an access proxy that's often cheaper than the VPN it replaces. There's a point of scale where a dedicated ZTNA platform earns its cost. Most regulated SMBs are years away from that point and would get more security per dollar finishing phases one and two.

The bar isn't "enterprise zero-trust." The bar is "a phished credential on an unmanaged device can't reach the PHI store." That bar is reachable this quarter, and it's the one that actually keeps you out of a breach notification.

## Frequently asked questions

**Q: Do SMBs really need to buy an enterprise zero-trust platform?**

No. Zero-trust is a principle — never trust, always verify — not a product, and a regulated SMB can reach roughly 80% of the benefit with identity tools it likely already owns. There's a scale where a dedicated ZTNA platform earns its cost, but most regulated SMBs are years away from it.

**Q: What's the first step to zero-trust for an SMB?**

Consolidate identity. That's the whole game and most of the value: SSO fronting every SaaS tool that supports it, MFA on 100% of human accounts, phishing-resistant MFA (FIDO2 or passkeys) on every admin, and named accounts replacing the shared logins so offboarding is a single revocation action.

**Q: Why isn't a VPN enough for zero-trust?**

A VPN is a moat, and the moat model assumes the attacker is outside. But the most reliable path in is a phished employee credential, and once it's inside a flat network the attacker reaches everything. Zero-trust replaces 'connect to the VPN and you're trusted' with per-application access that verifies identity and device every session.

**Q: In what order should the four zero-trust phases be done?**

Identity, then device, then network, then continuous verification. The order matters more than the tooling because each phase makes the next meaningful — buying the microsegmentation product before consolidating identity is spending on phase three while phase one is still open.

**Q: How much does zero-trust cost for a regulated SMB?**

Far less than the six-figure platform pitch. The four phases get you a genuine posture using an identity provider you already have, a modest MDM, cloud-native segmentation (VPC boundaries, security groups), and an access proxy like Cloudflare Access or Tailscale that's often cheaper than the VPN concentrator it replaces.

**Q: What's the realistic 'good enough' bar for SMB zero-trust?**

Not enterprise zero-trust — the bar is that a phished credential on an unmanaged device can't reach the PHI store. That's reachable this quarter with tools you mostly already own, and it's the bar that actually keeps you out of a breach notification.

Template: "Zero-trust phased maturity model" — branded PDF at https://kaansystems.com/templates/zero-trust-for-smbs.pdf

Source: https://kaansystems.com/library/zero-trust-for-smbs · Work with Kaan Systems: https://kaansystems.com/contact

---

# The 30-Day Cloud Cost Cleanup

Canonical: https://kaansystems.com/library/30-day-cloud-cost-cleanup
Topic: FinOps · Published: 2026-05-12 · Updated: 2026-10-12
Summary: Cloud bills compound. By year two of any AWS account, 20-40% of the monthly spend is on resources nobody remembers naming. A structured 30-day cleanup pulls it back without touching application code.

## Key takeaways

- A structured 30-day cleanup — inventory, eliminate, right-size, commit — typically recovers 20-40% of monthly cloud spend without touching a single application code path.
- Week 2's eliminations (idle EBS volumes, unattached Elastic IPs, orphaned snapshots, idle load balancers, stale AMIs and ECR images, and 24/7 dev environments) are pure waste and recover 5-15% of the bill on their own.
- Week 3 right-sizing leans on AWS Compute Optimizer for EC2 and RDS, Lambda Power Tuning for functions, a universal gp2 → gp3 migration, and ARM/Graviton where compatible — roughly 10-20% recovery.
- Week 4 locks in the savings: Compute Savings Plans covering 70-80% of baseline EC2/Fargate/Lambda, one-year no-upfront RDS Reserved Instances (~30% off list), and Spot for interruptible workloads (70-90% off on-demand).
- Cost work fails from the absence of a plan, not the absence of technique — the four-week structure is what turns identified opportunities into actual reductions.
- Run the cleanup quarterly and document each round's before/after numbers, because environments re-accumulate waste and cost reductions get forgotten faster than cost increases.

**How do you cut a cloud bill without touching application code?** Run a structured four-week cleanup: inventory what you're paying for (week 1), delete idle resources (week 2), right-size what's left (week 3), then commit to steady-state usage with Savings Plans (week 4). Most neglected environments recover 20-40% of monthly spend this way.

Week 3 is [rightsizing](/library/rightsizing-aws-without-breaking-production). The tags that make week 1 possible are [cost-aware tagging](/library/cost-aware-tagging). Week 4 is [Savings Plans versus Reserved Instances](/library/savings-plans-vs-reserved-instances). The [7-day Assessment](/assessment) names the waste before you commit.

## Where does a cloud bill hide its waste?

Cloud cost optimization generates a lot of conference talks and very little change. Most cost work fails not because the techniques are wrong but because there's no plan. The cost team identifies opportunities. Engineering disagrees about priority. Finance asks for a number by Friday. Nothing structural improves.

A 30-day plan changes that. Four weeks, four targeted activities. By the end, you've cut 20-40% of monthly spend in most environments without touching a single application code path.

## What happens in each of the four weeks?

```mermaid
flowchart LR
  W1["<b>WEEK 1</b><br/>Inventory<br/><br/>Tagging audit<br/>Top-10 cost-by-service<br/>Cost → workload map"]
  W2["<b>WEEK 2</b><br/>Eliminate<br/><br/>Idle EBS · EIPs · snapshots<br/>Idle ALBs · AMIs · ECR<br/>Dev shutdown schedules"]
  W3["<b>WEEK 3</b><br/>Right-size<br/><br/>Compute Optimizer triage<br/>EC2 · RDS gp3 · Lambda<br/>ARM where compatible"]
  W4["<b>WEEK 4</b><br/>Commit<br/><br/>Savings Plans signed<br/>RIs · Spot adoption<br/>MoM spend verified"]
  END(["20–40% spend recovered<br/>no app code touched"])

  W1 --> W2 --> W3 --> W4 --> END

  classDef week fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE
  classDef done fill:#1F4A3E,stroke:#3FA585,stroke-width:2.5px,color:#E8F5EE
  class W1,W2,W3,W4 week
  class END done
```

### Week 1: Inventory

You can't cut what you can't see. Week 1 is purely investigative.

**Day 1-2: Tag what's missing.** Run a tagging audit across every account. Anywhere a resource has no `owner`, `cost-center`, or `environment` tag, find the responsible team. If no team owns it, mark it as a candidate for elimination. Most environments find 10-15% of resources are untagged, and a meaningful fraction of those are abandoned.

**Day 3-4: Pull the cost-by-service report.** AWS Cost Explorer, broken down by service for the trailing 90 days. Print it. Highlight the top 10 services by spend. Those ten lines are 80%+ of your bill. Anything you don't recognize on that list is an investigation.

**Day 5: Map cost to service.** For each top-10 service, identify which application or workload is responsible. If you can't map a cost line to a workload, that's a finding. Usually a forgotten experiment, a deprecated product line, or a misconfigured backup.

Week 1 output: a tagged inventory, a cost-by-service report, and a list of resources or services with no clear owner.

### Week 2: Eliminate

The easiest cost savings are resources you can simply turn off. Order matters: handle the obvious cases first so you have momentum and signal for the team.

**Idle EBS volumes.** Run a query for any EBS volume in `available` state (not attached to any instance). These are 100% waste. Snapshot them if they're under 90 days old; delete the volume. Common AWS environments find a few percent of EBS spend in idle volumes.

**Unattached Elastic IPs.** AWS charges per hour for EIPs that aren't attached to a running instance. Find them, release them. Negligible per-EIP cost, but the principle matters: zero-tolerance for paying for nothing.

**Orphaned snapshots.** Snapshots whose source volume has been deleted, or snapshots more than two years old with no recent restore. The retention policy you'd set today is "keep the last 30 days plus the last day of every month for a year." Apply it retroactively.

**Idle load balancers.** ALBs and NLBs with zero traffic for 30 consecutive days. Either find why they exist, or delete them. The cost is small per-LB but the count grows fast.

**Old AMIs and ECR images.** AMIs older than 90 days that aren't referenced by any launch template. ECR images older than 180 days that aren't deployed anywhere. Both accumulate silently.

**Dev environments running 24/7.** Schedule dev/staging accounts to shut down outside business hours. Most teams find dev environments can run 50 hours/week instead of 168 with no impact on engineer velocity.

Week 2 output: a list of resources deleted, with rough monthly savings per category. Realistic recovery in this week alone: 5-15% of total bill.

### Week 3: Right-size

Now the work gets more careful. You're modifying running infrastructure.

**EC2 instances.** AWS Compute Optimizer surfaces over-provisioned instances based on CloudWatch metrics. For any instance running below 20% sustained CPU and below 40% sustained memory, propose a one-size-down change. Roll out one at a time during low-traffic windows; monitor for a week before declaring success.

**RDS instances.** Same idea, but cautiously. Compute Optimizer covers RDS too. The riskier change is gp2 → gp3 storage. Almost always strictly cheaper and faster, but requires a maintenance window.

**Lambda memory.** Teams over-provision Lambda memory because it's the only lever for CPU. Use the AWS Lambda Power Tuning tool to find the cost-optimal memory setting for each function. Often 256MB or 512MB outperforms 1024MB on both cost and latency.

**EBS volume types.** Migrate gp2 to gp3 universally. gp3 is faster, lets you provision IOPS independently of size, and is roughly 20% cheaper at the same provisioned size. There is no legitimate reason to launch a new gp2 volume.

**ARM where compatible.** AWS Graviton (ARM) instances are ~20% cheaper than x86 equivalents and often faster for many workloads. Anywhere your code runs on a managed runtime (Lambda, Fargate, App Runner) the switch is one config change. For self-managed instances, the work is rebuilding the AMI, which is a week of effort that recovers itself in two months.

Week 3 output: a list of rightsizings made, with metrics confirming workload health. Realistic recovery: 10-20%.

### Week 4: Commit

By week four, the workload mix has stabilized. Now you commit to it.

**Savings Plans.** Use the AWS Cost Explorer recommendations engine to identify the "Compute Savings Plan" coverage that matches your steady-state spend on EC2, Fargate, and Lambda. The standard recommendation is to cover 70-80% of your baseline; the remaining 20-30% stays flexible for growth or migration.

**Reserved Instances for RDS.** RDS doesn't have Savings Plans (yet); RIs are the equivalent. One-year, no-upfront commitments give ~30% off list with minimal lock-in risk.

**Spot instances for burst.** If you have batch workloads or any compute that can tolerate interruption, run them on spot. Discounts of 70-90% off on-demand.

**Sanity-check the savings.** Pull next month's projected spend versus this month. Confirm it dropped. If it didn't, your savings work was offset by growth. That's fine, but document it so the cost story is clear to leadership.

Week 4 output: signed Savings Plans, RI purchases, a projected-spend chart showing the reduction.

## The Template

Below is the weekly checklist. Each row is a single action with a clear "done" state. Run through it sequentially.

**Week 1: Inventory**
- Tagging audit complete; untagged resources flagged
- Cost-by-service top-10 list produced
- Cost-to-workload mapping complete; orphan workloads identified

**Week 2: Eliminate**
- All idle EBS volumes deleted (after snapshot)
- All unattached EIPs released
- Snapshots older than retention policy deleted
- Load balancers with zero traffic for 30+ days investigated or deleted
- AMIs / ECR images outside retention deleted
- Dev environment shutdown schedules in place

**Week 3: Right-size**
- Compute Optimizer recommendations triaged
- EC2 right-sizings rolled out (10+ instances)
- RDS gp2 → gp3 migrations complete
- Lambda memory tuned on top-spend functions
- ARM migration assessed for compatible workloads

**Week 4: Commit**
- Savings Plan signed for steady-state EC2/Fargate/Lambda
- RDS RIs purchased for steady-state databases
- Spot adopted for batch / interruptible workloads
- Month-over-month spend reduction confirmed

## Operating Notes

Run the cleanup quarterly, not once. The work compounds in both directions: environments accumulate waste, and your team forgets which knobs they turned. A 30-day cleanup every Q1 and Q3 keeps the bill close to its true workload cost.

Document the savings. Cost reductions get forgotten faster than cost increases. A one-page memo per cleanup, with the before/after numbers and the specific actions taken, builds the institutional memory that lets the next cleanup go faster.

The hard part is never the technique. It's getting four weeks of focused work on a problem that doesn't show up in product roadmaps.

## Frequently asked questions

**Q: How much can a 30-day cloud cost cleanup actually save?**

In environments that haven't been cleaned, a focused four-week cleanup typically recovers 20-40% of monthly spend — roughly 5-15% from eliminating idle resources, 10-20% from right-sizing, and the rest from commitment discounts. None of it requires changing application code.

**Q: Do I need to change application code to cut my cloud bill?**

No. The entire 30-day plan — deleting idle resources, right-sizing instances, migrating storage, and buying commitments — operates on infrastructure, not code. That is the point: it recovers 20-40% without touching a single application code path.

**Q: What's the fastest cloud waste to eliminate first?**

Idle EBS volumes sitting in the `available` state, unattached Elastic IPs, orphaned snapshots, and load balancers with zero traffic for 30 days. These are resources you can simply turn off, and week 2 handles them before you touch any running infrastructure.

**Q: Should I migrate from gp2 to gp3 EBS volumes?**

Yes. gp3 is faster, lets you provision IOPS independently of size, and is roughly 20% cheaper at the same provisioned size. The article's stance is blunt: there is no legitimate reason to launch a new gp2 volume.

**Q: How much of my baseline should Savings Plans cover?**

Cover 70-80% of your steady-state spend on EC2, Fargate, and Lambda with a Compute Savings Plan, leaving the remaining 20-30% flexible for growth or migration. For RDS, which has no Savings Plans, use one-year no-upfront Reserved Instances for about 30% off list.

**Q: How often should I run a cloud cost cleanup?**

Quarterly, not once. Environments accumulate waste and teams forget which knobs they turned, so a cleanup every Q1 and Q3 keeps the bill close to its true workload cost. Document each round's before/after numbers to build the institutional memory that speeds up the next one.

Template: "Weekly cost-cleanup checklist" — branded PDF at https://kaansystems.com/templates/30-day-cloud-cost-cleanup.pdf

Source: https://kaansystems.com/library/30-day-cloud-cost-cleanup · Work with Kaan Systems: https://kaansystems.com/contact

---

# Choosing Between SOC 2, HITRUST, and HIPAA Attestation

Canonical: https://kaansystems.com/library/choosing-soc2-hitrust-hipaa
Topic: Compliance · Published: 2026-05-12 · Updated: 2026-10-12
Summary: SOC 2, HITRUST, and HIPAA all sound like they certify 'security,' but they signal different things to different buyers. The decision framework for which one to pursue when.

## Key takeaways

- Which framework you need depends on your primary buyer, not on which one sounds most rigorous: SOC 2 for enterprise SaaS, HITRUST for hospital systems and health plans, a HIPAA assessment for smaller healthcare buyers.
- SOC 2 Type II is what enterprise buyers actually want: an independent auditor's opinion that your controls operated effectively over a 6-12 month period, costing $20-60K per cycle.
- HITRUST CSF r2 is the dominant security signal in healthcare procurement: a prescriptive checklist of 60-100+ controls costing $80-150K, taking 12-18 months, and reassessed every two years.
- HIPAA has no certification of its own; the Security Rule requires a compliance program, so 'HIPAA attestation' means a third-party assessment mapping your controls to the rule, usually packaged with SOC 2 or HITRUST.
- Don't run all three cycles at once. The control artifacts overlap about 80%, but the audit cycles and assessor relationships are distinct, and simultaneous cycles usually slip in teams under 100 engineers.
- The first framework you obtain is the expensive one; subsequent frameworks reuse the same policies and evidence pipelines, so plan SOC 2 to HITRUST as roughly half the work of SOC 2 alone.

**SOC 2, HITRUST, or HIPAA — which do you need?** Start with your primary buyer. Enterprise SaaS buyers want SOC 2 Type II ($20-60K). Hospital systems and health plans want HITRUST r2 ($80-150K). HIPAA has no certification of its own; only third-party assessments map to it. Most US healthcare-adjacent SaaS start with SOC 2, then add HITRUST when hospital deals appear.

If the buyer is a health system, the AWS question underneath the framework choice is [which services are HIPAA-eligible](/library/hipaa-eligible-aws-services) and whether you have [a minimum HIPAA posture](/library/minimum-viable-hipaa-aws) those services can sit in.

## Which framework is the buyer actually asking for?

A growth-stage company in healthcare or healthcare-adjacent SaaS reaches a point where customers start asking for security certifications. The CTO asks the question: SOC 2, HITRUST, HIPAA — which one do we need?

The answer is usually "it depends on who you sell to," which is unsatisfying. A more useful version of that answer below.

## How do SOC 2, HITRUST, and HIPAA differ?

The three frameworks are not interchangeable. Each one was designed to signal something specific, and the right choice depends on what your buyers are signaling for.

```mermaid
flowchart TD
  Q[Who is your primary buyer?]
  Q --> HOSP{Hospital systems<br/>or health plans?}
  Q --> ENT{Enterprise SaaS<br/>non-healthcare?}
  Q --> MIX{Mixed: healthcare<br/>+ enterprise?}
  Q --> GLOBAL{Significant<br/>EU customers?}

  HOSP -- yes --> HITRUST[Pursue HITRUST r2 first]
  ENT  -- yes --> SOC2[Pursue SOC 2 Type II first]
  MIX  -- yes --> SOC2HIPAA[SOC 2 Type II<br/>+ HIPAA mapping]
  GLOBAL -- yes --> SOC2ISO[SOC 2 + ISO 27001]

  HITRUST --> NEXT1[Add SOC 2 when<br/>non-hospital deals appear]
  SOC2 --> NEXT2[Add ISO 27001<br/>if going global]
  SOC2HIPAA --> NEXT3[Add HITRUST when<br/>healthcare deals grow]
  SOC2ISO --> NEXT4[Add HITRUST if<br/>healthcare in mix]

  classDef question fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px;
  classDef first    fill:#1F4A3E,stroke:#3FA585,stroke-width:2px;
  classDef next     fill:#0a1612,stroke:#2E8B6F,stroke-width:1px;

  class Q,HOSP,ENT,MIX,GLOBAL question;
  class HITRUST,SOC2,SOC2HIPAA,SOC2ISO first;
  class NEXT1,NEXT2,NEXT3,NEXT4 next;
```

### SOC 2

A SOC 2 report is an auditor's opinion on whether your organization meets the AICPA's Trust Services Criteria. There are five criteria: Security (always required), Availability, Processing Integrity, Confidentiality, and Privacy (all optional). Most reports cover Security plus one or two others.

**Two report types:** Type I is a point-in-time snapshot ("on January 1, these controls were designed and in place"). Type II is an operating-period assessment ("from January 1 through December 31, these controls were operating effectively"). Type II is what enterprise buyers actually want; Type I is a stepping stone.

**What it proves:** That your company has thought about security in a structured way, has documented controls, and an independent auditor confirmed the controls operate as documented over a multi-month period.

**Who asks for it:** Enterprise SaaS buyers. Procurement teams. Cybersecurity insurance underwriters. Investors during diligence.

**Cost:** $20-60K per cycle for a Type II report, plus internal time. The first cycle is the expensive one (3-6 months of readiness work plus the audit period); subsequent cycles are roughly half.

**Timeline:** 6-12 months from "we want SOC 2" to delivering a Type II report. The bottleneck is the operating period: the auditor needs to see controls running for at least 6 months (often 12) before they can issue Type II.

### HITRUST

The HITRUST CSF (Common Security Framework) is a prescriptive control set developed specifically for healthcare. It synthesizes HIPAA, NIST 800-53, ISO 27001, and several other frameworks into a single integrated control framework with three levels: r2 (rigor 2, the most common), r1 (rigor 1, lighter), and e1 (essentials).

**What it proves:** Your security posture meets the specific control set healthcare organizations expect from their vendors. HITRUST is much more prescriptive than SOC 2: a checklist of 60-100+ controls (depending on level) that you either implement or you don't.

**Who asks for it:** Hospital systems. Health plans (payers). Healthcare data exchanges. The HITRUST stamp is the dominant security signal in healthcare procurement.

**Cost:** $80-150K for an r2 certification cycle, plus significantly more internal time than SOC 2. Reassessment every 2 years.

**Timeline:** 12-18 months for first certification. The long pole is control implementation: most organizations need months of work to close the gaps before the assessment.

### HIPAA "Attestation"

Important clarification: **HIPAA itself has no certification.** The Security Rule says you must have a "compliance program." It does not say you must obtain a certificate of compliance. There is no government-issued HIPAA certification.

What exists are *third-party assessments* that map your controls to HIPAA Security Rule requirements. Common forms:

- **HIPAA Security Rule Compliance Assessment** by a third-party auditor produces a report you can share with customers.
- **HITRUST CSF certification** includes HIPAA mappings, so HITRUST is often used as HIPAA evidence.
- **SOC 2 + HIPAA.** Some auditors offer a combined report that overlays HIPAA mappings on a SOC 2 Type II.

**What it proves:** That you've engaged seriously with the HIPAA Security Rule and an independent assessor has reviewed your controls against it.

**Who asks for it:** Healthcare buyers who haven't standardized on HITRUST. Smaller healthcare organizations. Business associate agreements often require attestation of some kind.

**Cost:** Highly variable. $10-30K for a standalone HIPAA assessment; usually packaged with another framework.

**Timeline:** 3-6 months for an assessment cycle.

### ISO 27001

A related framework you might encounter: ISO 27001. International, prescriptive ISMS (Information Security Management System) certification. Popular with EU buyers and global enterprises.

**Cost:** $40-100K for first certification; 3-year cycle with annual surveillance.

**Timeline:** 6-12 months for first certification.

Skip this unless your sales motion includes significant European or global enterprise customers. For US-only healthcare or fintech, SOC 2 carries equivalent or stronger signal at lower cost.

## The Decision Matrix

Based on your primary buyer, here's where to start:

| Primary buyer | First framework to pursue | Second framework (if expanding) |
|---|---|---|
| Mid-market SaaS buyers | SOC 2 Type II | ISO 27001 (if going global) |
| Hospital systems | HITRUST r2 | SOC 2 (for non-hospital buyers) |
| Health plans / payers | HITRUST r2 | SOC 2 |
| Federal agencies buying a cloud product (SaaS, PaaS, IaaS) | FedRAMP, now under the FedRAMP 2026 Consolidated Rules (20x certification) | SOC 2 (for commercial buyers) |
| Federal agency systems you build or operate for the agency | Agency authorization (ATO) against NIST SP 800-53 | SOC 2 (for commercial buyers) |
| State Medicaid programs | Varies by state: read the state's contract and security requirements | HITRUST or a HIPAA assessment where the state accepts one |
| Healthcare-adjacent SaaS (selling to providers) | HITRUST r2 OR HIPAA assessment + SOC 2 | — |
| General enterprise + occasional healthcare | SOC 2 Type II with HIPAA mapping | HITRUST (if healthcare grows) |
| Global enterprise (EU + US) | SOC 2 + ISO 27001 | Add HITRUST if healthcare in mix |

The most common sequencing for a US healthcare-adjacent SaaS: [SOC 2 first](/library/soc2-for-startups-complete-guide) (to cover the largest buyer pool), then HITRUST when healthcare deals start losing on "you don't have HITRUST."

## Timeline & Cost Summary

Rough numbers for first-cycle:

| Framework | First-year cost | Internal time | Time to first report | Renewal cadence |
|---|---|---|---|---|
| SOC 2 Type II | $30-60K | 200-400 hours | 9-12 months | Annual |
| HITRUST r2 | $80-150K | 400-800 hours | 12-18 months | Every 2 years |
| HIPAA assessment | $10-30K | 100-200 hours | 3-6 months | Annual or biennial |
| ISO 27001 | $40-100K | 300-500 hours | 9-12 months | 3-year cycle + annual surveillance |

Internal time estimates assume a competent engineering team and one designated compliance owner. Without a dedicated owner, multiply by 2-3x; the work expands to fill the calendar.

## The Template

Use this cheat-sheet to make the decision:

**Step 1:** Identify your top 3 active deals (or top 3 target accounts).

**Step 2:** For each, find their security requirements in writing. Either ask the buyer, or look at their published vendor security expectations.

**Step 3:** Tally which frameworks appear:
- SOC 2: ____ deals
- HITRUST: ____ deals
- HIPAA assessment: ____ deals
- ISO 27001: ____ deals
- Other: ____ deals

**Step 4:** Apply the rule: pursue the framework that unblocks the most current revenue. If the tally is unclear, default to SOC 2 first — it has the broadest applicability and the most reusable artifacts.

**Step 5:** Set a realistic timeline. Most teams over-commit on the first cycle. Add 30% buffer to the auditor's estimate.

**Step 6:** Budget for it — both dollars and internal time. The single most common reason compliance projects fail is that internal time gets reallocated to product work mid-cycle.

## Operating Notes

Don't pursue all three frameworks at once. The artifacts overlap heavily (a control documented for SOC 2 is 80% reusable for HITRUST), but the audit cycles and assessor relationships are distinct. Trying to run two cycles simultaneously in a team of fewer than 100 engineers usually means both slip.

The first framework you obtain is the expensive one. Subsequent frameworks share the readiness work — the policies, the evidence pipelines, the security training program, the vendor management process all carry forward. Plan as if SOC 2 → HITRUST is half the work of SOC 2 alone.

The framework you have is less important than the operating muscle you build around it. The team that meaningfully runs SOC 2 — quarterly access reviews, monthly evidence collection, annual control updates — is the team that will pass HITRUST easily. The team that scrambled to pass SOC 2 once and forgot about it will fail HITRUST and then fail SOC 2 renewal.

Compliance is a continuous practice, not a project.

## Frequently asked questions

**Q: Do I need SOC 2, HITRUST, or HIPAA?**

It depends on your primary buyer. Enterprise SaaS buyers ask for SOC 2 Type II; hospital systems and health plans ask for HITRUST r2; smaller or non-standardized healthcare buyers accept a HIPAA assessment. Pursue whichever unblocks the most current revenue, and default to SOC 2 if the signal is mixed.

**Q: Is there a HIPAA certification?**

No. HIPAA's Security Rule requires you to have a compliance program, not a certificate, and there is no government-issued HIPAA certification. What you can obtain is a third-party assessment mapping your controls to the Security Rule, or you can use HITRUST or a combined SOC 2 + HIPAA report as evidence.

**Q: How much do SOC 2, HITRUST, and HIPAA cost?**

First-cycle ballpark: SOC 2 Type II runs $20-60K, HITRUST r2 $80-150K, a standalone HIPAA assessment $10-30K, and ISO 27001 $40-100K. Internal time is separate and often larger, from 100 to 800 hours depending on framework, and 2-3x that without a dedicated compliance owner.

**Q: Should I get SOC 2 or HITRUST first?**

For most US healthcare-adjacent SaaS, start with SOC 2 because it covers the largest buyer pool, then add HITRUST once healthcare deals start losing on 'you don't have HITRUST.' If hospital systems or health plans are your primary buyer from the start, pursue HITRUST r2 first.

**Q: Can I pursue all three frameworks at once?**

Not recommended. The control artifacts overlap heavily (a control documented for SOC 2 is about 80% reusable for HITRUST), but the audit cycles and assessor relationships are separate, and running two cycles at once in a team under 100 engineers usually means both slip.

**Q: Do I need ISO 27001?**

Skip it unless your sales motion includes significant European or global enterprise customers. For US-only healthcare or fintech, SOC 2 carries an equivalent or stronger signal at lower cost; add ISO 27001 only when you go global.

Template: "Framework selection cheat-sheet" — branded PDF at https://kaansystems.com/templates/choosing-soc2-hitrust-hipaa.pdf

Source: https://kaansystems.com/library/choosing-soc2-hitrust-hipaa · Work with Kaan Systems: https://kaansystems.com/contact

---

# Evidence Pipelines 101: How to Stop Reconstructing Audit Artifacts Quarterly

Canonical: https://kaansystems.com/library/evidence-pipelines-101
Topic: Compliance · Published: 2026-05-12 · Updated: 2026-10-12
Summary: Most teams reconstruct audit evidence from scratch every quarter. That's weeks of senior engineering time spent producing artifacts a running system could generate itself. The pipeline below does it automatically.

## Key takeaways

- Audits ask you to prove a control was operating across the entire period under review, not just designed once — and that operational evidence is exactly what teams reconstruct from scratch every cycle.
- Rebuilding evidence by hand runs about three engineers for two weeks each cycle, and it recurs because the underlying data rotates out of console retention before the next audit.
- Five categories of events cover roughly 80% of what any audit or customer review asks for: identity, change, configuration snapshots, data-access, and secrets and credential lifecycle.
- The architecture is identical for all five categories: source, a collector, durable storage in S3 Object Lock, an optional hash or signature, and a query layer auditors use directly.
- Keep raw evidence in S3 Object Lock for the full retention period (often seven years for HIPAA-aligned work), hold 18 months in a hot queryable layer, and rehydrate older data on demand.
- Once the pipeline runs, auditor questions become queries and the same store doubles as your incident-investigation timeline instead of a screenshot reconstruction.

**What is an evidence pipeline?** It's the system that generates audit artifacts continuously from your running infrastructure, so you stop rebuilding them under deadline every quarter. Collect five categories of events into durable, tamper-evident storage behind a query layer, and the next audit becomes one engineer for two days, not three for two weeks.

## Why do teams rebuild audit evidence every quarter?

Audits, customer security reviews, and regulatory inquiries all ask the same question in slightly different words: **show me that this control was operating during the period under review.**

The naive first-cycle answer is "let me put together a spreadsheet." Three engineers, two weeks, screenshot evidence pulled from a dozen consoles, manually attested to. The auditor accepts it because the alternative is missing the audit window. Their report dings the company for "compensating controls" or "manual evidence collection."

Six months later, the next cycle starts. The spreadsheet has to be rebuilt because the underlying data rotated out of console retention. Three engineers, two more weeks.

That's work automation should be doing.

[SOC 2 readiness](/library/soc2-for-startups-complete-guide) is where this bites first: the audit doesn't want to see the control once, it wants evidence it operated across the entire period, and that evidence is exactly what teams end up reconstructing every cycle.

## How do Terraform and CI produce that evidence instead?

Treat evidence as a continuously generated output of Terraform, CI, and CloudTrail, not a quarterly project. Those three streams already answer the change and configuration questions. Five categories of events still cover roughly 80% of what any audit or customer review will ask for. The same pattern is [drift detection](/library/infrastructure-drift-detection) for the plan diff, [GitOps](/library/gitops-progressive-delivery) for the deploy record, and the [platform engineering guide](/library/platform-engineering-for-regulated-startups), which is the Terraform hub. The [7-day Assessment](/assessment) includes a one-page evidence map of what those systems already prove.

### 1. Identity events

Every authentication, every authorization decision, every privileged action.

Sources: Cognito / Okta / Auth0 logs for human auth; CloudTrail (or equivalent) for API-level IAM events; Kubernetes audit log for cluster-level RBAC decisions.

What you'll be asked to prove: "Show me the privileged actions performed by terminated employees in the 30 days before their termination." Without an identity-events pipeline, this question takes a week to answer. With one, it's a query.

### 2. Change events

Every change to configuration, infrastructure, or production code.

Sources: GitHub / GitLab events (PRs merged), Terraform Cloud / Atlantis logs (infrastructure applies), AWS Config (resource configuration changes), CI/CD pipeline events.

What you'll be asked to prove: "Show me every production change in the period, who authored it, and who approved it." If your change-management process is mature, this is one query against three tables.

### 3. Configuration snapshots

Point-in-time snapshots of the security-relevant configuration of every resource.

Sources: AWS Config for AWS resources; Conftest / OPA snapshots for Kubernetes; vendor APIs for SaaS configurations.

What you'll be asked to prove: "Show me that on December 31, every S3 bucket holding PHI had encryption enabled and public access blocked." Config snapshots are exactly the right shape for this.

### 4. Data-access events

Every read of sensitive data.

Sources: S3 server access logs (or CloudTrail data events for the buckets that matter), RDS audit logs, application-layer audit logging.

What you'll be asked to prove: "Show me everyone who accessed the patient_records table in the period, what they queried, and what the business purpose was." Application-layer audit logging is what makes the third part of that question answerable; without it, you can prove access but not purpose.

### 5. Secrets and credential lifecycle events

Creation, rotation, access, and revocation of every credential.

Sources: AWS Secrets Manager / HashiCorp Vault audit log; certificate authority audit log for mTLS; SSH key rotation logs if applicable.

What you'll be asked to prove: "Show me that every production credential in this scope has been rotated within the last 90 days, and that no terminated employee retained access." A secrets pipeline answers this in one query.

## Implementation

The architectural shape is the same for all five categories:

```mermaid
flowchart LR
  subgraph SRC[Sources]
    direction TB
    S1[CloudTrail / Okta<br/>identity events]
    S2[GitHub / Terraform<br/>change events]
    S3[AWS Config<br/>config snapshots]
    S4[RDS audit / app audit<br/>data-access events]
    S5[Secrets Manager<br/>credential lifecycle]
  end

  COL[Collector<br/>EventBridge → SQS → Lambda<br/>or Kinesis Firehose]

  subgraph STORE[Durable storage]
    direction TB
    RAW[(S3 raw<br/>Object Lock · 7yr)]
    PARQ[(Parquet partitioned<br/>hot · 18 mo)]
    HASH[Daily SHA-256 manifest<br/>tamper-evident]
  end

  Q[Query layer<br/>Athena · BigQuery · Snowflake]
  AUDITOR((Auditor<br/>or incident<br/>investigator))

  S1 & S2 & S3 & S4 & S5 --> COL
  COL --> RAW
  COL --> PARQ
  RAW --> HASH
  PARQ --> Q
  RAW -.->|rehydrate on demand| Q
  Q --> AUDITOR

  classDef src   fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
  classDef pipe  fill:#1F4A3E,stroke:#3FA585,stroke-width:2px,color:#E8F5EE;
  classDef store fill:#0a1612,stroke:#2E8B6F,stroke-width:1.5px,color:#E8F5EE;
  classDef out   fill:#3FA585,stroke:#3FA585,stroke-width:2px,color:#0a1612;
  classDef band  fill:#2E8B6F0A,stroke:#2E8B6F,stroke-width:1px;

  class S1,S2,S3,S4,S5 src;
  class COL,HASH pipe;
  class RAW,PARQ store;
  class Q,AUDITOR out;
  class SRC,STORE band;
```

1. **Source.** Wherever the events are natively emitted.
2. **Collector.** A pipeline that pulls or receives the events and writes them to durable storage. For AWS, typically CloudTrail → Kinesis Firehose → S3, or EventBridge → SQS → Lambda → S3.
3. **Storage.** S3 with Object Lock (compliance mode) for raw evidence, plus a partitioned Parquet representation for query speed.
4. **Hash chain or signature.** Optional but worth doing: each batch is signed or hashed-and-chained so tampering is detectable. The simple version is daily SHA-256 manifests checked into a separate tamper-evident store.
5. **Query layer.** Athena, BigQuery, Snowflake, or whatever your team already uses. Auditors get a query interface, not a console screenshot.

The lifecycle policy is important. Raw evidence stays in S3 Object Lock for the full retention period (often seven years for HIPAA-aligned work). The hot, queryable layer holds 18 months. Anything older is rehydrated from cold storage on demand.

## The Template

The map below ties controls to evidence sources. Run through it for your environment; anywhere you can't fill in a source, query, or retention value is a gap to close before your next audit.

| Control area | Evidence type | Source | Sample query | Retention |
|---|---|---|---|---|
| Identity — privileged access | Auth + assumption events | CloudTrail `AssumeRole`, Identity Center login | `WHERE eventName='AssumeRole' AND user_arn LIKE '%admin%'` | 7 years |
| Identity — terminations | Account deactivation events | Identity Center, HR system join | `WHERE event='UserDeactivated' AND timestamp > <termination_date>` | 7 years |
| Change management — production | Merged PR + approver list | GitHub Audit Log | `WHERE action='pull_request.merged' AND repo IN <prod_repos>` | 7 years |
| Change management — infrastructure | Terraform apply events | TF Cloud / Atlantis | `WHERE event='terraform.apply' AND workspace LIKE 'prod-%'` | 7 years |
| Configuration — S3 encryption | Resource config snapshots | AWS Config | `SELECT resourceId WHERE configuration.encryption.status='Disabled'` | 7 years |
| Configuration — SG rules | Resource config snapshots | AWS Config | `SELECT resourceId WHERE configuration.ipPermissions[*].cidrIp='0.0.0.0/0'` | 7 years |
| Data access — PHI tables | DB audit log | RDS audit, application audit | `WHERE table_accessed IN <phi_tables> GROUP BY user` | 7 years |
| Data access — S3 PHI buckets | Object-level events | CloudTrail data events | `WHERE eventSource='s3.amazonaws.com' AND requestParameters.bucketName=<phi_bucket>` | 7 years |
| Secrets — rotation | Secret rotation events | Secrets Manager | `WHERE event='SecretRotated' AND age_days > 90` | 7 years |
| Secrets — credential revocation | Credential deletion events | IAM access keys, app secrets | `WHERE event='AccessKeyDeleted'` | 7 years |

Fill in the queries with your environment's specifics. Fill in the retention column based on your regulatory mix. The format is what matters: control mapped to source mapped to query mapped to retention.

## What This Buys You

The first audit after this is in place feels different. Instead of three engineers and two weeks, it's one engineer and two days. The auditor stops asking "can you produce this?" and asks "show me the query." You demo. They check the retention. Move on.

The second audit feels like maintenance. The third audit, the auditor stops asking because the answers have all been pre-generated.

The same pipeline doubles as your incident-investigation substrate. The next security incident, your team doesn't reconstruct the timeline from console screenshots. They query it.

## Frequently asked questions

**Q: Why do teams rebuild audit evidence from scratch every quarter?**

The first-cycle answer is usually a manually assembled spreadsheet of screenshots pulled from a dozen consoles. Six months later the underlying data has rotated out of console retention, so the spreadsheet has to be rebuilt — three engineers, two more weeks — every cycle.

**Q: What is an evidence pipeline?**

It treats audit evidence as a continuously-generated output of your running systems rather than a quarterly project, collecting security-relevant events into durable, tamper-evident storage behind a query layer. Auditors get a query interface instead of a console screenshot.

**Q: What kinds of evidence do audits and security reviews actually ask for?**

Five categories cover roughly 80% of the requests: identity events, change events, configuration snapshots, data-access events, and secrets and credential lifecycle events. Each maps to specific log sources like CloudTrail, GitHub, AWS Config, RDS audit logs, and Secrets Manager.

**Q: How long do you have to retain audit evidence?**

Keep raw evidence in S3 Object Lock for the full retention period, often seven years for HIPAA-aligned work. A hot, queryable layer holds 18 months, and anything older is rehydrated from cold storage on demand.

**Q: How do you make the evidence tamper-evident?**

Store raw evidence in S3 Object Lock in compliance mode, and sign or hash-and-chain each batch so tampering is detectable. The simple version is a daily SHA-256 manifest checked into a separate tamper-evident store.

**Q: How much does a working pipeline actually save at audit time?**

The first audit after it's in place drops from roughly three engineers and two weeks to one engineer and two days. By the third cycle the auditor stops asking because the answers are pre-generated, and the same pipeline becomes your incident-investigation substrate.

Template: "Evidence map (control → log source → query → retention)" — branded PDF at https://kaansystems.com/templates/evidence-pipelines-101.pdf

Source: https://kaansystems.com/library/evidence-pipelines-101 · Work with Kaan Systems: https://kaansystems.com/contact

---

# Detecting Infrastructure Drift Without Drowning in Alerts

Canonical: https://kaansystems.com/library/infrastructure-drift-detection
Topic: Platform · Published: 2026-05-12 · Updated: 2026-10-12
Summary: Drift detection at scale produces thousands of 'differences,' most of them noise. A tiered classifier routes the dangerous drift to PagerDuty and sends everything else to a weekly digest.

## Key takeaways

- Drift detection fails not because detection is hard but because the signal-to-noise ratio is brutal — out of roughly 200 drift entries, maybe two are worth a human's attention.
- Score drift into three tiers at detection time: Tier 1 (authorization, audit, availability) pages the on-call, Tier 2 (capacity, cost, consistency) goes to a weekly digest, and Tier 3 (known-good) is suppressed at the detector.
- Tier 1 covers IAM, security groups, KMS, CloudTrail, S3 policies, backup retention, and production load-balancer changes — drift there means either an attacker moving laterally or a bypassed change-control process.
- Suppressing Tier 3 is what makes the whole system usable; without it the digest is unreadable and the Tier 1 alerts get ignored.
- The classifier is regex over the terraform plan JSON — roughly 30 lines of Python mapping resource types to tiers, with no machine learning and no fancy heuristics.
- Watch how often Tier 1 fires: more than once a week on average signals a change-control gap, and less than once a quarter means the classifier is too lenient.

**How do you detect infrastructure drift without drowning in alerts?** Score every drift entry at detection time into three risk tiers: authorization, audit, and availability changes page the on-call; capacity and cost changes go to a weekly digest; and known-good changes get suppressed. The fix isn't better detection. It's classification, and a regex classifier does the routing.

The same plan signal shows up once you have a [multi-account landing zone](/library/multi-account-aws-landing-zone). The guides around it live on the [Terraform hub](/library/terraform). A drift scan is part of the [7-day Assessment](/assessment).

## Why does drift detection turn into alert noise?

Drift detection sounds great in principle and becomes useless in practice. Run `terraform plan` against production once a day and you'll get a delta. Run it across 80 accounts and you'll get a delta the length of a novel.

The teams that turn it on usually turn it off within a month. The signal-to-noise ratio is brutal: out of two hundred drift entries, maybe two represent something you'd actually want a human to look at. The other 198 are autoscaling resizing a node pool, a managed service updating its own backing IAM policy, or a tag your costing system added after the fact.

The fix isn't better detection. It's classification.

## How do you tier drift so only the dangerous changes page?

Treat drift as a continuously scored risk surface, not as a binary state mismatch. Score it at detection time into three tiers:

```mermaid
flowchart LR
  D[terraform plan<br/>detects drift]
  D --> CL{Classifier<br/>regex over plan.json}

  CL -- "IAM, SG, KMS,<br/>CloudTrail, S3 policy" --> T1[Tier 1<br/>Auth · audit · availability]
  CL -- "instance type, tags,<br/>ASG bounds, Lambda mem" --> T2[Tier 2<br/>Capacity · cost · consistency]
  CL -- "ASG counts, AWS-managed<br/>policies, service-linked roles" --> T3[Tier 3<br/>Known-good — suppressed]

  T1 --> PD[PagerDuty<br/>page the on-call · hourly]
  T2 --> SL[Slack digest<br/>weekly review · Mondays]
  T3 --> NULL["/dev/null"]

  classDef src    fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
  classDef gate   fill:#1F4A3E,stroke:#3FA585,stroke-width:2px,color:#E8F5EE;
  classDef hot    fill:#3FA585,stroke:#3FA585,stroke-width:2px,color:#0a1612;
  classDef warm   fill:#1F4A3E,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
  classDef cold   fill:#0a1612,stroke:#2E8B6F,stroke-width:1px,color:#A8B8B0;

  class D src;
  class CL gate;
  class T1,PD hot;
  class T2,SL warm;
  class T3,NULL cold;
```

### Tier 1: Authorization, audit, and availability drift

Changes that, left alone, change who can do what, what gets logged, or what happens when something fails. Every entry in this tier pages the on-call:

- IAM policies, roles, or trust relationships
- Security group rules (especially anything opening 0.0.0.0/0)
- KMS key policies or rotation settings
- CloudTrail or VPC Flow Log configuration
- S3 bucket policies, public access blocks, encryption settings
- Backup plan deletion or retention reduction
- Production load-balancer listener changes

Drift in any of these means either an attacker is moving laterally or a human bypassed your change-control process. Either is worth waking someone up.

### Tier 2: Capacity, cost, and consistency drift

Not dangerous on their own, but indicators of a process gap. These go to a weekly digest:

- Instance type changes (someone resized for cost)
- Tag additions or removals (cost-allocation drift)
- Auto-scaling group min/max adjustments
- RDS parameter group tweaks
- Lambda memory or timeout adjustments

Once a week, an engineer reviews the digest. If a pattern is real ("the data team is rightsizing RDS by hand every Friday"), the IaC catches up. If it's one-off ("someone changed a memory limit during an incident"), the engineer either backports or accepts.

### Tier 3: Known-good drift

Expected changes you don't want cluttering the digest. Suppress them at the detector:

- Autoscaling instance counts inside declared bounds
- AWS-managed IAM policy version updates
- Service-linked role updates
- Default route table associations created by service launches
- CloudFront distribution metric subscriptions

Suppressing Tier 3 is what makes the system usable. Without it, the digest is unreadable and Tier 1 alerts get ignored.

## Implementation

Tooling: `terraform plan` in CI on a schedule is the baseline. For scale, `driftctl` or Pulumi's `pulumi refresh --diff` produce the same signal with less compute.

Run the detector hourly for Tier 1 (cheap, narrow scope), daily for Tier 2 (broader plan), weekly for the full estate. Route Tier 1 to PagerDuty via a webhook that includes the resource ARN and the field that changed. Route Tier 2 to a Slack channel. Route Tier 3 to `/dev/null`.

The classifier is regex on the planned change. Five lines of Python wraps the terraform plan JSON output into a tier judgment. No machine learning. No fancy heuristics. A static table maps `aws_iam_*` and `aws_security_group_rule` and a few others to Tier 1; everything else gets reviewed by the digest.

## The Template

Below is the GitHub Actions workflow that runs the hourly Tier 1 check against an estate of AWS accounts. It uses an OIDC-federated role to assume into each account, runs `terraform plan -detailed-exitcode`, and routes anything matching the Tier 1 pattern to a webhook.

```yaml
# .github/workflows/drift-check.yml
name: Infrastructure drift · Tier 1
on:
  schedule:
    - cron: '0 * * * *'   # hourly
  workflow_dispatch:

permissions:
  id-token: write          # for OIDC role assumption
  contents: read

jobs:
  detect:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        account: [prod-us-east-1, prod-us-west-2, log-archive]
      fail-fast: false
    steps:
      - uses: actions/checkout@v4

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::${{ secrets.ACCOUNT_ID }}:role/DriftDetector
          aws-region: us-east-1

      - uses: hashicorp/setup-terraform@v3
        with: { terraform_version: 1.9.x }

      - name: terraform plan
        id: plan
        working-directory: ./environments/${{ matrix.account }}
        run: |
          terraform init -input=false
          terraform plan -input=false -lock=false \
            -out=plan.bin -detailed-exitcode || exit_code=$?
          terraform show -json plan.bin > plan.json
          echo "exit_code=${exit_code:-0}" >> $GITHUB_OUTPUT

      - name: Classify and route
        if: steps.plan.outputs.exit_code == '2'   # changes detected
        run: |
          python3 .ci/classify-drift.py plan.json > tier1.json
          if [ -s tier1.json ]; then
            curl -X POST -H 'Content-Type: application/json' \
              --data @tier1.json \
              ${{ secrets.PAGERDUTY_WEBHOOK }}
          fi
```

The `classify-drift.py` script is a 30-line filter over the JSON plan that emits Tier 1 entries only:

```python
import json, sys, re

TIER1_RESOURCES = (
    r'^aws_iam_',
    r'^aws_security_group',
    r'^aws_kms_key',
    r'^aws_cloudtrail',
    r'^aws_s3_bucket_policy$',
    r'^aws_s3_bucket_public_access_block$',
    r'^aws_backup_(plan|vault)',
)

plan = json.load(open(sys.argv[1]))
tier1 = []

for change in plan.get('resource_changes', []):
    if change['change']['actions'] == ['no-op']:
        continue
    if any(re.match(p, change['type']) for p in TIER1_RESOURCES):
        tier1.append({
            'resource': change['address'],
            'type': change['type'],
            'actions': change['change']['actions'],
        })

if tier1:
    json.dump({'alert': 'Tier-1 drift', 'changes': tier1}, sys.stdout)
```

## Operating Notes

Run the digest review on Mondays. Anything in Tier 2 that's been there three consecutive weeks becomes a Tier 1 candidate, or gets accepted and moved to Tier 3 with a comment explaining why. The tiers aren't static. They're the team's working classification of what matters, and they should evolve as you learn what your environment actually does.

The number to watch isn't drift incidents. It's how often Tier 1 fires. More than once a week on average and your change-control process has a gap. Less than once a quarter and the classifier is too lenient. Someone is changing security-relevant settings without it surfacing. Drift detection is one guardrail in the broader [platform engineering foundation](/library/platform-engineering-for-regulated-startups) a regulated startup runs; treat this classifier as that foundation's early-warning system.

## Frequently asked questions

**Q: How do you stop drift detection from becoming alert noise?**

Classify drift by risk at detection time instead of treating every difference as an alert. Route dangerous changes (auth, audit, availability) to PagerDuty, send capacity and cost drift to a weekly digest, and suppress known-good drift entirely. The detection tooling is mostly solved; the classifier is the actual work.

**Q: What counts as Tier 1 drift that should page the on-call?**

Changes to IAM policies and roles, security group rules (especially anything opening 0.0.0.0/0), KMS key policies, CloudTrail or VPC Flow Log config, S3 bucket policies and public-access blocks, backup retention, and production load-balancer listeners. Drift in any of these means an attacker is moving laterally or someone bypassed change control.

**Q: How often should you run infrastructure drift detection?**

Run the narrow Tier 1 check hourly because it's cheap, run a broader Tier 2 plan daily, and run the full estate weekly. Route Tier 1 to PagerDuty, Tier 2 to a Slack digest reviewed on Mondays, and Tier 3 to /dev/null.

**Q: Do you need machine learning to classify infrastructure drift?**

No. The classifier is regex over the terraform plan JSON — about 30 lines of Python that map resource types like aws_iam_* and aws_security_group to Tier 1 and send everything else to the digest. No machine learning, no fancy heuristics.

**Q: How do you know if your drift classifier is tuned correctly?**

Watch how often Tier 1 fires. More than once a week on average means your change-control process has a gap; less than once a quarter means the classifier is too lenient and security-relevant changes are slipping through unsurfaced.

**Q: What tools can detect infrastructure drift?**

terraform plan on a schedule in CI is the baseline. At scale, driftctl or Pulumi's pulumi refresh --diff produce the same signal with less compute. The template here uses a GitHub Actions workflow with an OIDC-federated role and terraform plan -detailed-exitcode.

Template: "Terraform drift-check GitHub Actions workflow" — branded PDF at https://kaansystems.com/templates/infrastructure-drift-detection.pdf

Source: https://kaansystems.com/library/infrastructure-drift-detection · Work with Kaan Systems: https://kaansystems.com/contact

---

# A Minimum-Viable HIPAA Compliance Posture for AWS

Canonical: https://kaansystems.com/library/minimum-viable-hipaa-aws
Topic: Compliance · Published: 2026-05-12 · Updated: 2026-10-29
Summary: Most HIPAA-on-AWS guides are overwhelming. The minimal posture that passes audit and lets engineering keep moving is twelve controls, grouped across account architecture, identity, data protection, and observability.

## Key takeaways

- HIPAA on AWS reduces to twelve technical controls across four areas: account architecture, identity, data protection, and observability.
- Sign the AWS Business Associate Addendum in AWS Artifact first, and keep PHI on HIPAA-eligible services only (~150 today) — treat eligibility as a runtime check, not a one-time decision.
- Separate workloads, logs, and audit tooling into at least three AWS accounts, enforced with Organizations SCPs; the highest-leverage SCP denies CloudTrail deletion.
- Encrypt PHI at rest with KMS and enforce TLS 1.2+ in transit; AWS-managed keys are fine for the minimum-viable posture.
- Ship an org-wide CloudTrail to a log-archive account with S3 Object Lock (compliance mode) so the logs are tamper-proof — even to root.
- SIEM, GuardDuty, Macie, and WAF are a should, not a must: layer them on when the audit cycle starts to hurt.

## What does enough HIPAA on AWS actually require?

Most HIPAA-on-AWS guides try to be exhaustive and end up unactionable. Two hundred controls drawn from the full Security Rule, with no help separating "you must have this" from "you should have this eventually."

Engineering teams in early-stage healthcare companies don't have time for two hundred controls. They need the minimal technical posture that satisfies the Security Rule without delaying the product roadmap by a quarter.

There is one.

## Which controls are the minimum, and where do they live?

HIPAA on AWS reduces to twelve technical controls grouped into four areas. Have these in place and you have a defensible technical posture. Administrative and physical safeguards layer on top later, but auditors ask about the technical layer first, and engineering owns it end-to-end. Most of these controls do double duty for [SOC 2 readiness](/library/soc2-for-startups-complete-guide) — the encryption, access, and tamper-proof-logging requirements overlap almost entirely — so if a SOC 2 audit is next, the HIPAA work carries straight over.

### Sign the BAA, then use only eligible services

Before anything else, sign the AWS Business Associate Addendum from the AWS Artifact console. The BAA is what makes AWS a HIPAA business associate; without it, you have no legal coverage for storing or transmitting PHI on the platform.

The BAA only covers AWS's defined list of "HIPAA-eligible services" (currently around 150). Use a service outside that list to handle PHI and you've stepped outside the BAA. The list shifts over time, so treat eligibility as a runtime check, not a one-time decision.

### Account architecture: separate workloads from logs from audit

Run PHI workloads in a dedicated AWS account. Run a separate account for centralized logs. Run a third account as the audit/security tooling landing zone. The three-account split is the minimum that gives you the four properties HIPAA cares about:

1. Production-data access is narrow and traceable.
2. Logs cannot be tampered with by the same identities that can touch production.
3. Audit tooling has a stable home with its own access boundary.
4. Account-level blast radius is contained.

Use AWS Organizations with Service Control Policies (SCPs) to enforce account boundaries. The single highest-leverage SCP is "deny removal of CloudTrail in any account."

```mermaid
flowchart LR
  subgraph ORG[AWS Organization · SCPs enforce boundaries]
    direction LR

    subgraph WL[Workload account]
      direction TB
      APP[Application + PHI data]
      RDS[(RDS · KMS-encrypted)]
      S3W[(S3 · PHI · KMS)]
      APP --> RDS
      APP --> S3W
    end

    subgraph LOG[Log-archive account]
      direction TB
      CT[(CloudTrail logs)]
      VPC[(VPC Flow Logs)]
      OL[S3 Object Lock<br/>compliance mode]
      CT --> OL
      VPC --> OL
    end

    subgraph AUD[Audit / security account]
      direction TB
      CFG[AWS Config]
      GD[GuardDuty]
      IAM[Identity Center]
    end

    WL -.->|"CloudTrail trail<br/>(org-wide)"| LOG
    WL -.->|"Resource snapshots"| AUD
    AUD -.->|"Read-only audit"| WL
    AUD -.->|"Read-only audit"| LOG
  end

  classDef wl  fill:#1F4A3E,stroke:#3FA585,stroke-width:2px,color:#E8F5EE;
  classDef log fill:#0d1f1a,stroke:#3FA585,stroke-width:2px,color:#E8F5EE;
  classDef aud fill:#0a1612,stroke:#2E8B6F,stroke-width:1.5px,color:#E8F5EE;
  classDef band fill:#2E8B6F0A,stroke:#2E8B6F,stroke-width:1px;
  class APP,RDS,S3W wl;
  class CT,VPC,OL log;
  class CFG,GD,IAM aud;
  class ORG,WL,LOG,AUD band;
```

### Identity: MFA, no root, scoped roles

Three rules: every human has MFA on their console access; the root account is locked away and not used for anything except billing; humans assume cross-account roles via IAM Identity Center (formerly AWS SSO) rather than holding long-lived IAM users.

Service identities use IAM roles with the narrowest possible scope. If your CI pipeline needs to deploy to S3 and invalidate a CloudFront distribution, its role gets exactly those two actions. Not `s3:*`. Not `cloudfront:*`.

### Data: encrypt at rest, encrypt in transit, manage your keys

Everything storing PHI must be encrypted at rest with AWS KMS. The default AWS-managed key is acceptable for the MVP posture; customer-managed keys (CMKs) with annual rotation become valuable when you're large enough that key-access auditing matters.

In transit: TLS 1.2 minimum on every endpoint. Disable HTTP entirely at the load-balancer level. mTLS for service-to-service is worth doing eventually, but TLS termination at the load balancer with private VPC routing is enough for the MVP.

### Observability: log everything, then make the logs untouchable

CloudTrail organization-wide trail logging to the log-archive account, S3 destination configured with Object Lock in compliance mode. Object Lock is what turns "logs you keep" into "logs an auditor will believe." It prevents anyone, including the AWS root user, from deleting logs within the retention period.

VPC Flow Logs into the same archive account. Application logs into CloudWatch Logs with one-year retention minimum. The MVP retention rule: longer than your average incident-discovery window plus six months. Six years is HIPAA's general recommendation; one year is the operational floor.

### Backup: it's part of compliance, not separate from it

AWS Backup with cross-region copy for any service holding PHI. The Security Rule's contingency-plan requirement calls for both backup procedures and disaster recovery. A documented annual restore drill closes the loop on both.

[Book a call →](https://cal.com/benrubert/30min?utm_source=library&utm_medium=article&utm_campaign=minimum-viable-hipaa-aws&utm_content=middle)

## The Template

The twelve-control map below is what the MVP posture looks like, scored against your environment. Each row gets a `yes` / `no` / `partial`. Where you have partial coverage, write a one-line note describing what's missing. The whole document fits on a page.

> **Want this scored against your estate?** The [7-day Assessment](/assessment) is read-only and covers the account split, logging, and encryption on this checklist.

| # | Area | Control | Service / mechanism |
|---|---|---|---|
| 1 | Legal | BAA signed | AWS Artifact |
| 2 | Legal | Only HIPAA-eligible services touch PHI | Documented list |
| 3 | Architecture | Workloads / logs / audit separated into 3+ accounts | AWS Organizations |
| 4 | Architecture | SCPs prevent CloudTrail removal | SCP `s-DenyCloudTrailDelete` |
| 5 | Identity | MFA required on all human console access | Identity Center policy |
| 6 | Identity | Root account locked + monitored | Alarm on root usage |
| 7 | Identity | Service roles least-privileged | IAM access analyzer |
| 8 | Data | All PHI data stores encrypted at rest | KMS |
| 9 | Data | All endpoints TLS 1.2+ | ELB / CloudFront / API Gateway |
| 10 | Observability | CloudTrail org-wide → log-archive account with Object Lock | CloudTrail + S3 Object Lock |
| 11 | Observability | VPC Flow Logs centralized | VPC + S3 |
| 12 | Backup | Cross-region backup + annual restore drill | AWS Backup |

Run through it for your environment. Anywhere you score `no`, decide one of three things: implement it this quarter, accept and document the gap, or escalate it as a compliance risk.

## What's Deliberately Not Here

The MVP skips: SIEM, intrusion detection (GuardDuty is a "should," not a "must"), Macie for PHI discovery, Web Application Firewall rules, third-party CSPM tooling. All add value. None are required for a defensible HIPAA posture on day one.

Layer them on when the audit cycle starts to feel painful. Until then, the twelve above are the work.

## Frequently asked questions

**Q: What is the minimum HIPAA setup on AWS?**

A defensible minimum is twelve technical controls across four areas: account architecture (separate workload, log-archive, and audit accounts plus SCPs), identity (MFA everywhere, locked root, least-privilege roles), data (KMS encryption at rest and TLS 1.2+ in transit), and observability (an org-wide CloudTrail shipped to a tamper-proof log archive). Sign the AWS BAA before any of it.

**Q: Do I need a BAA with AWS for HIPAA?**

Yes. Sign the AWS Business Associate Addendum in AWS Artifact before you store or transmit PHI. Without it you have no legal coverage, and it only extends to AWS's HIPAA-eligible services.

**Q: Which AWS services are HIPAA-eligible?**

Only the services AWS lists as HIPAA-eligible (around 150 today) are covered by the BAA; using anything else to handle PHI steps outside it. The list changes over time, so verify eligibility as a runtime check rather than trusting a one-time decision.

**Q: How many AWS accounts do I need for HIPAA?**

At least three: a workload account for PHI, a separate log-archive account, and an audit or security account, tied together with AWS Organizations SCPs. That split gives traceable production access, tamper-proof logs, and contained blast radius.

**Q: Is AWS-managed KMS enough for HIPAA, or do I need customer-managed keys?**

AWS-managed KMS keys are acceptable for a minimum-viable HIPAA posture. Customer-managed keys with rotation become worthwhile once you are large enough that key-access auditing matters.

**Q: What HIPAA controls can I skip at first?**

SIEM, GuardDuty, Macie, WAF, and third-party CSPM tooling all add value but are not required for a defensible day-one posture. Add them when the audit cycle starts to feel painful.

Template: "Control-mapping checklist" — branded PDF at https://kaansystems.com/templates/minimum-viable-hipaa-aws.pdf

Source: https://kaansystems.com/library/minimum-viable-hipaa-aws · Work with Kaan Systems: https://kaansystems.com/contact

---

# Architecture Decision Records: The 1-Page Version

Canonical: https://kaansystems.com/library/one-page-adrs
Topic: Engineering Practice · Published: 2026-05-12 · Updated: 2026-10-12
Summary: Architecture Decision Records are valuable. The canonical Michael Nygard format is also long enough that most teams don't keep up with writing them. The one-page version preserves the value with a tenth of the friction.

## Key takeaways

- The canonical Michael Nygard ADR format, dating to 2011, runs four to six pages — comprehensive, but long enough that most teams never actually write them.
- The one-page ADR is six fields — Title, Status, Context, Decision, Consequences, References — capped at a single screen, with no subsections and no formal status transitions.
- Status has exactly four values: Proposed, Accepted, Superseded by ADR-NNN, and Deprecated. The typical path is Proposed → Accepted, with a small fraction later marked Superseded.
- Write an ADR only when a decision is hard to reverse (more than a week of work to undo), has architectural consequences that aren't obvious from the code, or was contested before being made — skip library versions, implementation details, and anything the code already records.
- ADRs live in the repository under docs/decisions/ as numbered markdown files, so they're versioned alongside the code and survive Confluence migrations, Notion reorganizations, and tool changes.
- When an ADR is superseded, mark the status instead of deleting it — the original reasoning is what lets a future reader avoid recreating the same mistake.

**What is a one-page ADR, and why use it instead of the full format?** A one-page Architecture Decision Record captures a decision in six fields — Title, Status, Context, Decision, Consequences, References — on a single screen. It keeps everything that makes ADRs valuable and drops the ceremony of the four-to-six-page canonical format, so engineers actually write them.

Two decisions that belong in this format the week you make them: [Argo CD versus Flux](/library/argo-cd-vs-flux) and [Control Tower versus Terraform](/library/control-tower-vs-terraform-landing-zone). The [7-day Assessment](/assessment) is where those calls get made against the estate instead of in a thread.

## Why don't engineers write architecture decision records?

Architecture Decision Records are one of the best ideas in software engineering documentation. They capture *why* a system is built the way it is. Not what it does. What was decided, and what was rejected.

The problem is that the canonical Michael Nygard format, dating to 2011, runs four to six pages: full context, complete options analysis, multi-paragraph consequences, formal status with state transitions. Comprehensive. Useful as a reference. Almost never written.

Engineers don't write ADRs because:

- The format implies a level of formality that doesn't match how decisions actually get made.
- A four-page ADR feels disproportionate to most decisions.
- Reviewing a four-page ADR is its own effort.
- By the time you have time to write the full version, the decision is six weeks old and the context is gone.

The fix is a shorter format. The one-page ADR keeps every component that makes ADRs valuable, drops everything that's optional or ceremonial, and fits in a single screen.

## What fits on one page and still gets written?

Six fields. Maximum one screen. No subsections. No formal status transitions. No optional fields. The shape is the contract.

### Field 1: Title

Numbered, present tense. The number is monotonic across the repo's history (ADR-001, ADR-002, ...). The title states the decision affirmatively, not the question:

> **Bad:** ADR-007: How should we handle async jobs?
> **Good:** ADR-007: Use SQS + Lambda for asynchronous job processing

A reader scanning the index should be able to know the decision from the title alone.

### Field 2: Status

One of four values:

- **Proposed.** Under discussion. The ADR exists but the decision isn't made.
- **Accepted.** Decision made, no superseding decision yet.
- **Superseded by ADR-NNN.** A later ADR overrides this one. Both stay in the repo.
- **Deprecated.** The decision no longer applies (system retired, constraint changed) but no new decision replaces it.

That's the entire state machine. No "rejected." No "under review." No nesting. The typical path is Proposed → Accepted, with a small fraction eventually marked Superseded.

```mermaid
stateDiagram-v2
  direction LR
  [*] --> Proposed: ADR drafted
  Proposed --> Accepted: decision made
  Proposed --> [*]: discarded before merge
  Accepted --> Superseded: later ADR overrides
  Accepted --> Deprecated: context changed,<br/>no replacement
  Superseded --> [*]
  Deprecated --> [*]

  note right of Superseded
    Keep both ADRs.
    Future readers need
    the original reasoning.
  end note
```

### Field 3: Context

Three to four sentences. What's the situation that requires a decision? What forces are at play? No history lesson, no scope justification, no roadmap context.

> **Bad:** "Our system has been growing for two years. As traffic increased, we noticed that synchronous calls to our PDF rendering service started to time out. We had a meeting where we discussed several options including..."
> **Good:** "PDF rendering takes 5-30 seconds per request. Synchronous handling causes p95 latency violations and ties up application threads. We need an async path."

If you can't make the context fit in four sentences, the decision is probably actually two decisions.

### Field 4: Decision

One sentence. State the decision, not the deliberation.

> **Bad:** "After considering Lambda, ECS, and self-hosted workers, we decided that SQS + Lambda was the best option because..."
> **Good:** "Render PDFs via SQS + Lambda, with the queue fronted by an API Gateway endpoint."

The reasoning belongs in Consequences, not Decision. The Decision field is the answer; everything else is supporting.

### Field 5: Consequences

Three bullets. Positive, negative, neutral. Each bullet is one sentence.

> **Positive:** Application threads released immediately; horizontal scale is automatic.
> **Negative:** Lambda cold starts add 200-400ms to first-request latency.
> **Neutral:** Operational visibility shifts from APM to CloudWatch Logs + queue metrics.

Three bullets forces honesty. You can't write three meaningful positive bullets if the decision has only two real upsides, and you can't hide the downsides when the negative bullet has to be substantive.

### Field 6: References

Links. PRs that implement the decision, related ADRs, external documentation, design docs. Bulleted list, no annotations needed: the link text or URL is enough.

If there are no references, leave the field blank with the heading present (so the template stays consistent across ADRs).

## When to Write an ADR — and When Not To

```mermaid
flowchart TD
  Start[A decision was made]
  Q1{Hard to reverse?<br/>more than 1 week<br/>of work to undo}
  Q2{Architectural<br/>consequences not<br/>obvious from code?}
  Q3{Was it contested<br/>before being made?}
  Q4{Will it stabilize<br/>within 3 months?}

  Start --> Q1
  Q1 -- no --> SKIP[Skip — code is the record]
  Q1 -- yes --> Q2
  Q2 -- no --> Q3
  Q2 -- yes --> WRITE[Write the ADR]
  Q3 -- yes --> WRITE
  Q3 -- no --> Q4
  Q4 -- no --> WAIT[Wait — revisit when stable]
  Q4 -- yes --> WRITE

  classDef yes fill:#1F4A3E,stroke:#3FA585,stroke-width:2px;
  classDef no  fill:#0a1612,stroke:#2E8B6F,stroke-width:1px;
  classDef q   fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px;
  class Start,Q1,Q2,Q3,Q4 q;
  class WRITE yes;
  class SKIP,WAIT no;
```

Write one when:

- The decision is hard to reverse cheaply. "Hard" here means more than a week of engineering work to undo.
- The decision has architectural consequences that aren't obvious from the code alone.
- A future engineer reading the codebase will ask "why is it like this?"
- The decision was contested in any way before being made.

Don't write one for:

- Library versions or framework choices that you'd swap in a sprint.
- Implementation details (variable naming, function structure, file layout).
- Anything fully captured in code or config. The code is the record.
- Decisions you're going to revisit in three months. Wait until they stabilize.

The hardest rule to enforce is the first one. Engineers tend to write ADRs for every meaningful decision rather than only the consequential ones. That's a learnable habit, and the cost of one too many is much lower than the cost of skipping the one that mattered.

## Where ADRs Live

In the repository. Specifically, in `docs/decisions/`, as numbered markdown files: `0001-use-postgres.md`, `0002-async-jobs-via-sqs-lambda.md`, etc.

The repo is the right home because:

- The decisions are versioned alongside the code they describe.
- The Pull Request that implements the decision can reference the ADR by file path.
- A new engineer reading the codebase finds the ADRs in the natural place.
- The ADRs survive Confluence migrations, Notion reorganizations, and tool changes.

There are tools (adr-tools, log4brains) that automate ADR numbering and templating. They're helpful but optional. Markdown files in `docs/decisions/` work fine with no tooling at all.

## The Template

The single-page format, copy-pasteable into a new `docs/decisions/NNNN-short-title.md`:

```markdown
# ADR-NNNN: <Decision in present tense>

## Status

Proposed | Accepted | Superseded by ADR-NNNN | Deprecated

## Context

<3-4 sentences. What's the situation? What forces require a decision now?>

## Decision

<One sentence. The decision itself.>

## Consequences

- **Positive:** <one sentence>
- **Negative:** <one sentence>
- **Neutral:** <one sentence>

## References

- <PR link>
- <Related ADR>
- <External documentation>
```

That's the entire format. Six headings, four to six fields filled in, total length under a screen on a normal monitor.

## Operating Notes

Make ADRs part of the PR template for significant changes. The PR template includes a line like "Does this require an ADR? If yes, link the file." That single question catches the changes that should have one.

In code review, ask for ADRs the way you ask for tests: not for every change, but for changes where the absence is conspicuous. "Why is this a separate service?" usually means an ADR is missing.

When an ADR is superseded, *don't delete it.* Mark the status. The history is part of what makes ADRs valuable. Six months from now, someone reads ADR-007 (superseded), realizes the original reasoning, and avoids recreating the same mistake.

The first month after introducing the practice, expect the team to write three or four ADRs. The second month, fewer. They've learned what counts. By month six, you have a small library of decisions that captures the architectural skeleton of the system in a form that's still readable three years and three engineering generations later.

## Frequently asked questions

**Q: What is a one-page ADR?**

A one-page ADR is an Architecture Decision Record compressed to six fields — Title, Status, Context, Decision, Consequences, References — that fits on a single screen. It keeps everything that makes ADRs valuable and drops the ceremony of the canonical four-to-six-page format.

**Q: Why don't engineers write traditional ADRs?**

The canonical Michael Nygard format runs four to six pages, which feels disproportionate to most decisions and implies more formality than how decisions actually get made. By the time someone has time to write the full version, the decision is weeks old and the context is gone.

**Q: When should I write an ADR?**

Write one when a decision is hard to reverse (more than a week of work to undo), has architectural consequences that aren't obvious from the code, or was contested before being made. Don't write one for library versions, implementation details, or anything fully captured in code or config.

**Q: Where should ADRs be stored?**

In the repository, under docs/decisions/ as numbered markdown files like 0001-use-postgres.md. Keeping them with the code means they're versioned alongside it and survive Confluence and Notion migrations. Tools like adr-tools and log4brains help but are optional.

**Q: What are the four ADR statuses?**

Proposed, Accepted, Superseded by ADR-NNN, and Deprecated. That is the entire state machine — no rejected, no under review — and the typical path is Proposed → Accepted, with a small fraction eventually marked Superseded.

**Q: What happens to an ADR when the decision changes?**

Don't delete it. Mark the status as Superseded (or Deprecated) and keep both records in the repo. The original reasoning is part of the value: months later someone can read why the earlier decision was made and avoid recreating the same mistake.

Template: "1-page ADR markdown template" — branded PDF at https://kaansystems.com/templates/one-page-adrs.pdf

Source: https://kaansystems.com/library/one-page-adrs · Work with Kaan Systems: https://kaansystems.com/contact

---

# A Pragmatic Threat Model for Regulated SMBs

Canonical: https://kaansystems.com/library/pragmatic-threat-model
Topic: Security · Published: 2026-05-12 · Updated: 2026-10-12
Summary: Threat-modeling templates aimed at Google and Microsoft assume APT-level adversaries. Regulated SMBs mostly face commodity threats plus regulator scrutiny. The model below fits that reality.

## Key takeaways

- The canonical threat-modeling guides — STRIDE, OWASP, and the NIST publications — were written for organizations facing the full threat spectrum, including nation-state adversaries; a regulated SMB doesn't face that.
- By volume, roughly 95% of what hits any internet-exposed SMB is commodity criminal traffic: credential stuffing, phishing, and ransomware-as-a-service.
- Use STRIDE as the frame but model only three of its six categories deeply — spoofing, tampering, and information disclosure; repudiation, denial of service, and elevation of privilege are largely covered by those controls.
- Identity hardening is the single highest-ROI security work for an SMB: universal MFA, SSO consolidation, FIDO2 for admins, and regular access reviews.
- Regulators audit your documentation of controls, not the controls themselves — teams that lose HIPAA audits usually have controls but can't show them in writing.
- The one-page template scores three categories on residual risk (1-5) for an aggregate out of 15: above 9, don't launch; 7-9, launch with a documented mitigation plan and a 90-day review; below 7, launch.

**What does a pragmatic threat model for a regulated SMB look like?** One page, not sixty attack vectors. Regulated SMBs mostly face commodity criminal traffic, regulator scrutiny, and negligent insiders — not nation-state APTs. Use STRIDE as the frame, model the three high-priority categories (spoofing, tampering, information disclosure) deeply, and spend on identity, backups, vendors, and documentation.

## Whose threat model are you borrowing?

The canonical threat-modeling guides (STRIDE from Microsoft, the OWASP Threat Modeling Cheat Sheet, the various NIST publications) were written by and for organizations that face the full threat spectrum: nation-state adversaries, sophisticated criminal groups, hacktivist movements, supply-chain attackers with multi-year operational patience.

A regulated small-to-medium business in healthcare, legal services, or fintech doesn't face that spectrum. It faces:

- **Commodity criminal traffic.** Credential stuffing, phishing, ransomware-as-a-service. By volume, 95% of what comes at any internet-exposed business.
- **Regulator scrutiny.** Not strictly an adversary, but the highest-impact "external party reviewing your security posture" event.
- **Negligent insiders.** Well-meaning employees doing the wrong thing. Far more common than malicious insiders.
- **Opportunistic vulnerability scanning.** Automated scans looking for unpatched CVEs and misconfigurations.
- **Vendor security incidents.** Third parties whose compromise becomes your compromise.

The mismatch between the canonical threat-modeling guidance and the actual SMB threat profile produces threat models that are either too theoretical (worried about adversaries that won't show up) or too granular (cataloguing 60 attack vectors that all reduce to "patch your stuff and turn on MFA").

The pragmatic threat model is one page, identifies the four or five threats that actually matter, and maps them to specific controls.

## Which threats show up for a team this size?

STRIDE — the structure used by most threat modeling — has six categories. Three are high-priority for the SMB profile; three are lower-priority. Use STRIDE as the frame, but allocate modeling time according to that profile.

```mermaid
flowchart LR
  subgraph HIGH[High priority — model deeply]
    direction TB
    S[Spoofing<br/>credential theft, phishing, BEC]
    T[Tampering<br/>unauthorized prod changes]
    I[Information disclosure<br/>regulator-relevant breaches]
  end

  subgraph LOW[Lower priority — covered by HIGH controls]
    direction TB
    R[Repudiation<br/>audit logging from I]
    D[Denial of service<br/>cloud-native scaling + CDN]
    E[Elevation of privilege<br/>downstream of S]
  end

  HIGH -.-> LOW

  classDef high fill:#1F4A3E,stroke:#3FA585,stroke-width:2px,color:#E8F5EE;
  classDef low  fill:#0a1612,stroke:#2E8B6F,stroke-width:1px,color:#E8F5EE;
  classDef band fill:#2E8B6F0F,stroke:#2E8B6F,stroke-width:1px;
  class S,T,I high;
  class R,D,E low;
  class HIGH,LOW band;
```

### High-priority categories

**Spoofing.** Credential theft and identity impersonation. The SMB attacker's #1 path. Credential stuffing against your login forms, phishing of employee credentials, business email compromise.

Mitigations that matter:
- MFA universal (no exceptions, no SMS fallback if avoidable)
- SSO across SaaS (one identity to revoke, not 40)
- Adaptive auth on the login form (rate limiting, anomaly detection)
- Phishing-resistant MFA (FIDO2 / passkeys) for privileged users

**Tampering.** Modification of data or configuration without authorization. For SMBs this maps almost entirely to "unauthorized changes to production infrastructure" rather than data tampering, because SMB applications usually have proper data-layer controls.

Mitigations that matter:
- Change control with auditable trail (every prod change tied to a PR or ticket)
- Immutable infrastructure where possible
- Drift detection on the security-relevant subset (see [the drift detection guide](/library/infrastructure-drift-detection))
- Backups that can't be deleted by the same identities that can touch production (immutable backups)

**Information disclosure.** Unauthorized read access. For SMBs in regulated industries, this is the regulator-relevant category. HIPAA breaches, GDPR violations, breach-notification requirements all live here.

Mitigations that matter:
- Encryption at rest with customer-managed keys for the data that's regulator-relevant
- Encryption in transit (TLS 1.2+ everywhere)
- Access logs on every data store touching regulated data
- Quarterly access review (who has access, do they still need it)

### Lower-priority categories

**Repudiation.** Denial that an action took place. Less of an SMB concern; mostly applicable to financial-transaction systems. Standard mitigation is audit logging, which you already need for the higher-priority categories.

**Denial of service.** Resource exhaustion. SMBs are rarely the primary target for DDoS, but they're often collateral damage from infrastructure that wasn't built for elasticity. Cloud-native scaling plus a CDN with DDoS protection (Cloudflare, AWS Shield Standard) handles 95% of the SMB DoS risk surface.

**Elevation of privilege.** Gaining unauthorized higher access. Important, but mostly an output of the other categories. If spoofing is well-controlled, EoP rarely happens via remote attackers. The realistic SMB scenario is "an engineer's laptop gets compromised, attacker pivots to cloud admin." Defense is endpoint security plus workstation-to-cloud access proxies.

## Where to Actually Spend Effort

For the regulated SMB, four areas absorb most of the security investment. In rough order of impact-per-dollar:

1. **Identity hardening.** MFA universal, SSO consolidation, FIDO2 for admins, regular access reviews. The single highest-ROI security work for any SMB, and the same work as [killing long-lived credentials](/library/killing-long-lived-credentials).

2. **Backup and recovery.** Ransomware is the most likely catastrophic event. Immutable backups, restore drills, and a documented recovery posture turn ransomware from a business-ending event into a 48-hour outage.

3. **Vendor trust chain.** Your security floor is your suppliers' security floor. A vendor breach (think: Snowflake credentials in 2024, MOVEit in 2023) becomes your breach if you used the vendor with sensitive data. Vendor security review at procurement; annual reattestation for material vendors.

4. **Documentation hygiene.** Regulators don't audit your controls; they audit your documentation of your controls. Teams that lose HIPAA audits are rarely the ones without controls. They're the ones that have controls but can't show them in writing. Policies, procedures, runbooks, and an [evidence pipeline](/library/evidence-pipelines-101), kept current. The [7-day Assessment](/assessment) scores that list against the estate.

## The Template

The one-page threat model for an SMB system or feature looks like this. Fill it in for any new system before launch, and re-review annually.

**System / feature:** _____________________

**Owner:** _____________________

**Data sensitivity:** ☐ Regulated (PHI / PII / PCI) ☐ Confidential ☐ Internal ☐ Public

**Spoofing threats**
- Primary scenario: ______________________________
- Existing mitigation: ___________________________
- Residual risk (1-5): ____

**Tampering threats**
- Primary scenario: ______________________________
- Existing mitigation: ___________________________
- Residual risk (1-5): ____

**Information disclosure threats**
- Primary scenario: ______________________________
- Existing mitigation: ___________________________
- Residual risk (1-5): ____

**Aggregate risk score:** ____ / 15

**Risk acceptance:** Above 9 → don't launch. 7-9 → launch with documented mitigation plan + 90-day review. Below 7 → launch.

**Notes / open questions:** _____________________

**Sign-off:** _____________________ Date: ___________

Three categories, one page, scored honestly. The model takes 30 minutes to fill in for a new system, surfaces the threats that matter, and gives leadership a defensible record of what was considered before launch.

## What This Buys You

Two things.

**First, the security program becomes legible.** When a customer, an auditor, or a board member asks "how do you think about security?", you have a structured answer that doesn't hand-wave. The answer is "we threat-model every new system, here's the template, here are the last six completed models."

**Second, the team stops doing security theater.** No more two-hour threat-modeling sessions for a feature that ships in a week. No more 200-question threat assessments that nobody reads. One page, three categories, ship.

The first time the model surfaces a real risk, and it will, the model has paid for itself.

## Frequently asked questions

**Q: Do small businesses really need to worry about nation-state hackers?**

Mostly no. A regulated SMB faces commodity criminal traffic, regulator scrutiny, negligent insiders, opportunistic vulnerability scanning, and vendor security incidents — not the nation-state and sophisticated-criminal adversaries the canonical guides are built around. Model for the threats that actually show up.

**Q: Which threats should a regulated SMB model most deeply?**

Three STRIDE categories: spoofing (credential theft, phishing, business email compromise), tampering (unauthorized production changes), and information disclosure (regulator-relevant breaches). The other three — repudiation, denial of service, and elevation of privilege — are lower priority and mostly handled by the controls you build for the top three.

**Q: Where should an SMB actually spend its security budget?**

Four areas, in rough order of impact per dollar: identity hardening, backup and recovery, the vendor trust chain, and documentation hygiene. Identity hardening is the highest-ROI work; immutable backups turn ransomware from a business-ending event into a 48-hour outage.

**Q: How long does the one-page threat model take to complete?**

About 30 minutes for a new system. Fill it in before launch and re-review it annually. The point is to surface the threats that matter and give leadership a defensible record of what was considered before launch.

**Q: What's the risk-acceptance rule in the template?**

Score the three categories on residual risk from 1 to 5 for an aggregate out of 15. Above 9, don't launch. Between 7 and 9, launch with a documented mitigation plan and a 90-day review. Below 7, launch.

**Q: Why do teams with real controls still fail HIPAA audits?**

Because regulators audit your documentation of your controls, not the controls themselves. The teams that lose audits usually have the controls but can't show them in writing — so keep policies, procedures, runbooks, and evidence pipelines current.

Template: "One-page threat model template" — branded PDF at https://kaansystems.com/templates/pragmatic-threat-model.pdf

Source: https://kaansystems.com/library/pragmatic-threat-model · Work with Kaan Systems: https://kaansystems.com/contact

---

# What "Production-Ready" Actually Means for a Kubernetes Cluster

Canonical: https://kaansystems.com/library/production-ready-kubernetes
Topic: Platform · Published: 2026-05-12 · Updated: 2026-10-12
Summary: 'Production-ready' is one of those phrases that lets engineering and product teams nod at each other while meaning different things. The concrete checklist that resolves the ambiguity: five dimensions, forty items, scored 0-2.

## Key takeaways

- Production-ready is a checklist, not an inherent property — a cluster either has the capabilities or it doesn't, and the checklist makes that explicit and scoreable.
- The scorecard spans five dimensions — Identity & Access, Network, Workload Health, Observability, and Recovery — with eight items each scored 0, 1, or 2, for a maximum of 80 points.
- 70 or above is production-ready, 60-69 is production-soft (fine only for non-customer-facing or non-regulated workloads), and below 60 is a demo that shouldn't run production traffic or hold PHI.
- Most teams score around 50 on their first honest pass; the next quarter's work is moving that 50 to 70, which is where the architectural investment lives.
- Re-score every cluster every six months, because readiness drifts: items that once scored 2 slip to 1 when a person leaves, a process lapses, or a tool changes.
- Make the scorecard part of a cluster's launch criteria, and when a customer asks whether your platform is production-grade, share the methodology rather than the raw score.

**What does "production-ready" actually mean for a Kubernetes cluster?** It's a checklist, not a property. Score the cluster across five dimensions — identity and access, network, workload health, observability, and recovery — with eight items each, rated 0 to 2. Seventy or more out of eighty is production-ready. Below sixty is a demo.

## Why do teams disagree on what production-ready means?

A new Kubernetes cluster goes from "demo working" to "production-ready" through a series of capabilities that aren't visible in any individual feature. The cluster has them all, or it doesn't. The people responsible for declaring "production-ready" (engineering leadership, security review, a customer's procurement team) usually don't have a shared definition of what those capabilities are. Cluster readiness is one pillar of the [Terraform and platform guide](/library/platform-engineering-for-regulated-startups). The deploy path is [GitOps and progressive delivery](/library/gitops-progressive-delivery), the controller choice is [Argo CD vs. Flux](/library/argo-cd-vs-flux), and the reliability number the scorecard should meet is an [SLO](/library/slos-for-small-teams). The [7-day Assessment](/assessment) scores the same items.

That causes two failure modes:

- A cluster gets declared production-ready and isn't. The first incident reveals that nothing was set up to recover. The team learns the gap the expensive way.
- A cluster gets blocked from production indefinitely because no two reviewers can agree on what's missing. The launch slips by quarters.

The fix is a checklist. Not exhaustive. Concrete and scoreable.

## How do you score a cluster so the argument ends?

Five dimensions. Each has eight items. Each item scores 0, 1, or 2:

- **0.** Not addressed.
- **1.** Partially or informally addressed.
- **2.** Fully addressed and operationally verified.

A score of 70+/80 across the dimensions is "production-ready." 60-69 is "production-soft." Below 60 is "demo." The threshold is somewhat arbitrary. The score is what matters: a number leadership can use to compare clusters and track progress.

```mermaid
flowchart TB
  CL((Cluster))
  CL --- D1[Identity &amp; Access<br/>8 items · 0-16 pts]
  CL --- D2[Network<br/>8 items · 0-16 pts]
  CL --- D3[Workload Health<br/>8 items · 0-16 pts]
  CL --- D4[Observability<br/>8 items · 0-16 pts]
  CL --- D5[Recovery<br/>8 items · 0-16 pts]

  D1 --> T[Total · 0-80]
  D2 --> T
  D3 --> T
  D4 --> T
  D5 --> T

  T --> R1[70-80<br/>Production-ready]
  T --> R2[60-69<br/>Production-soft]
  T --> R3[Below 60<br/>Demo]

  classDef hub  fill:#1F4A3E,stroke:#3FA585,stroke-width:2.5px,color:#E8F5EE;
  classDef dim  fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
  classDef good fill:#1F4A3E,stroke:#3FA585,stroke-width:2px,color:#E8F5EE;
  classDef warn fill:#0d1f1a,stroke:#2E8B6F,stroke-width:1px,color:#E8F5EE;
  classDef bad  fill:#0a1612,stroke:#1F4A3E,stroke-width:1px,color:#A8B8B0;

  class CL,T hub;
  class D1,D2,D3,D4,D5 dim;
  class R1 good;
  class R2 warn;
  class R3 bad;
```

### Dimension 1: Identity & Access

The cluster's answer to "who can do what."

1. **Cluster-level authentication source** is an enterprise identity provider (OIDC, SAML), not static kubeconfigs.
2. **RBAC is least-privilege by default.** No default-admin bindings. No `cluster-admin` ClusterRoleBindings to groups larger than a handful of platform engineers.
3. **Service accounts are namespaced and tightly scoped.** No application uses the `default` service account.
4. **Workload identity** (IRSA on EKS, Workload Identity on GKE, AAD Pod Identity on AKS) ties pod-level access to cloud IAM. No long-lived cloud credentials in cluster.
5. **Audit logging** for the API server is on, shipped to a log-archive store outside the cluster, with at least 90-day retention.
6. **MFA enforced** on the identity provider for any user with cluster access.
7. **Break-glass access** is documented: who has it, how it's used, how it's audited after.
8. **No long-lived kubeconfigs** issued to anyone. Access is short-lived and re-authenticated.

### Dimension 2: Network

How traffic flows in, out, and between pods.

1. **NetworkPolicy enforcement enabled** on the cluster (most clusters require an explicit CNI plugin choice for this).
2. **Default-deny network policy** in every namespace; explicit allow rules for required traffic.
3. **Ingress controllers terminate TLS** at the edge with managed certificates that auto-renew (cert-manager + ACME is standard).
4. **Egress controlled.** Either a NAT gateway with logging, an egress proxy, or — for stricter postures — explicit egress NetworkPolicy.
5. **Service mesh or sidecar mTLS** for service-to-service traffic in regulated environments. Not mandatory for non-regulated workloads.
6. **DNS policy** doesn't allow arbitrary external DNS. CoreDNS is configured with explicit forwarders.
7. **Load-balancer health checks** properly configured (path, interval, threshold) — not the AWS default which often misses application-level health.
8. **No services exposed as LoadBalancer or NodePort** without explicit review. Ingress is the only sanctioned external entry path.

### Dimension 3: Workload Health

What happens when things start to fail.

1. **Every deployment has liveness and readiness probes**, distinct from each other and not the default "/healthz returns 200" placeholder.
2. **Resource requests and limits set** on every container. Not optional. Enforced via OPA/Gatekeeper or Kyverno.
3. **PodDisruptionBudgets** on every production workload, set to allow at most 25% disruption.
4. **Horizontal Pod Autoscaler or KEDA** configured for workloads with variable load.
5. **Anti-affinity rules** prevent all replicas of a critical workload from landing on the same node.
6. **Quality of Service class is Guaranteed** for critical workloads (requests = limits).
7. **Init containers have timeouts** and are not used for arbitrary one-time setup that should be a Job.
8. **Graceful shutdown** is implemented in application code. SIGTERM is handled; in-flight requests complete.

### Dimension 4: Observability

What the operator can see during normal operation and during incidents.

1. **kube-state-metrics** scraped by Prometheus or equivalent.
2. **Node-level metrics** (node-exporter, kubelet) scraped.
3. **Application metrics** exposed by every service, with at minimum the four Golden Signals (latency, traffic, errors, saturation).
4. **Centralized logging** ships every container's stdout/stderr to a queryable store with at least 30-day retention.
5. **Audit log retention** beyond the centralized logging system (separate retention rule for security events).
6. **SLOs defined** for every production service, with error budget tracking visible on a shared dashboard.
7. **Alerts route to a paging system** with documented runbooks. Pages are tested quarterly.
8. **Tracing** enabled for services in the request path (less critical for backend batch services).

### Dimension 5: Recovery

What happens when the cluster, or a piece of it, fails entirely.

1. **etcd backups** every 30 minutes, retained for 30 days, stored outside the cluster.
2. **Application data backed up** via Velero or equivalent, with cross-region copies for regulated environments.
3. **Restore drill performed at least annually.** Documented runbook with timestamps.
4. **Documented disaster-recovery posture.** RTO and RPO targets stated, mapped to actual backup cadence.
5. **HA control plane.** Managed services (EKS, GKE, AKS) provide this; self-managed clusters need explicit verification.
6. **Multi-AZ workload spread** for production deployments via anti-affinity or topology spread constraints.
7. **Stateful workloads use StorageClasses** with documented backup and snapshot policies.
8. **Cluster upgrade procedure documented and rehearsed.** Last upgrade was within the support window of the current Kubernetes version.

## How do you upgrade without a version-skew outage?

A cluster that scored well last year and has not been upgraded is not production-ready. Kubernetes allows only a limited skew between the control plane and the nodes, and EKS, AKS, and GKE drop minor versions on a published calendar. Move one minor at a time, nodes after the control plane, and rehearse the procedure on a non-production cluster first. The scorecard already asks whether the last upgrade sat inside the current version's support window. If it did not, that row is a zero even when every other row is a two. The Assessment checks version, skew, and whether anyone has run the upgrade since the last support boundary.

## The Template

Score your cluster row by row. Anywhere you can't honestly score 2, write a one-line note explaining the gap and the planned remediation.

Total score: __ / 80

- **70-80: Production-ready.** Document the score, the date, the cluster, and circulate to whoever needs to approve.
- **60-69: Production-soft.** OK for non-customer-facing or non-regulated workloads. Not OK for the production workload your customers depend on.
- **Below 60: Demo.** Don't run production traffic. Don't put PHI on it. Don't claim it as "production" in customer contracts.

The number you produce will feel low the first time. That's the point. Most teams score around 50 on the first honest pass. The next quarter's work is moving 50 to 70.

## Operating Notes

Re-score every cluster every six months. Drift happens. Items that scored 2 become 1 because a person left, a process lapsed, a tool changed. The score is a leading indicator.

When a new cluster is built, the scorecard is part of the launch criteria. Cluster cannot host production traffic until the score is documented and approved. This creates an explicit gate that engineering, security, and product can all point at.

When a customer asks "is your platform production-grade?" — share the scorecard methodology, not the raw score. The methodology builds trust; the raw score gets weaponized in contract negotiations.

## Frequently asked questions

**Q: What does 'production-ready' actually mean for a Kubernetes cluster?**

It means the cluster passes a concrete checklist rather than possessing some vague quality. You score it across five dimensions — identity and access, network, workload health, observability, and recovery — and a total of 70 or more out of 80 is production-ready.

**Q: How is the production-readiness scorecard structured?**

Five dimensions, eight items each, and every item scored 0 (not addressed), 1 (partially or informally addressed), or 2 (fully addressed and operationally verified). That gives each dimension 0-16 points and a cluster a total out of 80.

**Q: What score counts as production-ready?**

70 to 80 is production-ready, 60 to 69 is 'production-soft' — acceptable for non-customer-facing or non-regulated workloads but not for the production workload customers depend on — and below 60 is a demo that shouldn't carry production traffic or PHI. The threshold is somewhat arbitrary; the number's real value is comparing clusters and tracking progress.

**Q: Why did my cluster score so low the first time?**

That's expected. Most teams score around 50 on their first honest pass, and the honesty is the point. The next quarter's work is moving that 50 to 70, which is where the identity, network-policy, and observability investment lives.

**Q: How often should you re-score a cluster?**

Every six months. Readiness drifts — an item that scored 2 becomes a 1 when a person leaves, a process lapses, or a tool changes — so the score is a leading indicator rather than a one-time certificate.

**Q: Should you show the scorecard to customers?**

Share the methodology, not the raw score. The methodology builds trust; the raw number tends to get weaponized in contract negotiations. Making the scorecard part of a cluster's launch criteria also gives engineering, security, and product one explicit gate to point at.

Template: "Production-readiness scorecard" — branded PDF at https://kaansystems.com/templates/production-ready-kubernetes.pdf

Source: https://kaansystems.com/library/production-ready-kubernetes · Work with Kaan Systems: https://kaansystems.com/contact

---

# Reading an Enterprise Security Questionnaire

Canonical: https://kaansystems.com/library/reading-security-questionnaire
Topic: Sales · Published: 2026-05-12 · Updated: 2026-10-12
Summary: Enterprise security questionnaires aren't really about your security controls. They're about whether you've thought about each control well enough to answer honestly. Four answer patterns plus a template covering the 30 most common questions.

## Key takeaways

- An enterprise security questionnaire tests whether you've thought about each control well enough to answer honestly, not whether you've implemented every control perfectly; a Series A company isn't expected to match a Fortune 500's control posture.
- Every question maps to one of four honest answers: yes with brief evidence, yes with honest scope, a compensating control that addresses the same underlying risk, or a clear 'not applicable' with a one-line reason.
- Brevity wins because the CISO is reading hundreds of these, so a long answer to a simple question reads as defensiveness; never lie or oversell, because it resurfaces in clarification calls, renewals, or a breach investigation.
- Compensating-control answers are where smaller companies earn trust: name what you do instead and show the architectural choice was deliberate and its risk profile understood.
- Recognize the framework (SIG, CAIQ, VSAQ, or a custom variant) so you can reuse prior answers; the first questionnaire takes a week, the fifth takes a day, and the fiftieth is a copy-paste plus a delta review.
- Build a versioned response repository keyed by question, tag every answer with a 'last verified' date and re-verify anything older than six months, and give one person ownership of each response.

An enterprise security questionnaire isn't testing whether you've implemented every control perfectly. It's testing whether you've thought about each one well enough to answer honestly. Read each question into one of four answers: yes with evidence, yes with honest scope, a compensating control, or not-applicable with a reason. That's how you ship in days, not weeks.

## What is a long security questionnaire actually asking?

You're closing a six-figure enterprise deal. The prospect's procurement team sends a 240-question security questionnaire on a Wednesday afternoon, due Friday morning. The format is a spreadsheet. The questions range from "do you have a written information security policy?" to "describe your container orchestration platform's key rotation cadence."

A team that hasn't seen one of these before responds by panicking, scheduling a war room for Thursday, and shipping a half-baked response. The deal closes on Monday after a clarification call where the prospect's CISO walks the responder through the questions they answered wrong.

A team that's been through this twice responds differently. They allocate four hours, work through the questionnaire methodically, and ship a response that closes the deal without needing the clarification call.

The difference is reading the questionnaire correctly.

## How do you answer from the repo?

Enterprise security questionnaires test one thing primarily: **whether you've thought about each control well enough to give an honest answer.** Despite appearances, they're not testing whether you've implemented every control perfectly.

That changes how you answer. The CISO reading your response doesn't expect a Series A company to have the same control posture as a Fortune 500. They expect you to know your posture, describe it accurately, and explain what's compensating where coverage is incomplete.

Every question has one of four honest answers:

```mermaid
flowchart TD
  Q[Read the question]
  IMPL{Do you implement<br/>this control?}
  EXACT{Exactly as<br/>the question asks?}
  EQUIV{Do you do something<br/>that addresses the<br/>same underlying risk?}
  SURF{Does the capability<br/>even apply to your<br/>architecture?}

  Q --> IMPL
  IMPL -- yes --> EXACT
  IMPL -- no  --> EQUIV

  EXACT -- yes --> P1[Pattern 1<br/>Yes — brief + evidence]
  EXACT -- no  --> P2[Pattern 2<br/>Yes with honest scope]

  EQUIV -- yes --> P3[Pattern 3<br/>Compensating control]
  EQUIV -- no  --> SURF

  SURF -- yes --> P3
  SURF -- no  --> P4[Pattern 4<br/>Not applicable + why]

  classDef q       fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px;
  classDef pattern fill:#1F4A3E,stroke:#3FA585,stroke-width:2px;
  class Q,IMPL,EXACT,EQUIV,SURF q;
  class P1,P2,P3,P4 pattern;
```

### Pattern 1: Yes (we do this)

You have the control fully implemented and can show evidence. The right answer is brief, factual, and points to evidence:

> Q: Do you encrypt customer data at rest?
> A: Yes. All customer data is encrypted at rest using AES-256 via AWS KMS. Customer-managed keys are available on request. Evidence: SOC 2 Type II report, section 4.2.

Don't oversell. Don't pad. The CISO is reading 240 of these. They will reward brevity. A long answer to a simple question reads as defensiveness.

### Pattern 2: Yes with caveats

You do the thing, but not the way the question implies. State the affirmative; clarify the scope:

> Q: Do you maintain a 24/7 SOC?
> A: We maintain 24/7 on-call coverage with a 30-minute response SLO for security incidents. We do not currently run a dedicated SOC team; security monitoring is integrated into the platform engineering on-call rotation, with automated detection routed to the same paging system as availability incidents. Evidence: on-call rotation policy, sample incident timeline.

Don't say "no" when the answer is "yes, but smaller in scope." Don't say "yes" when the implementation is materially different. The middle ground (affirmative, with honest scope) is what enterprise CISOs are looking for.

### Pattern 3: Compensating control

You don't do the thing the question asks, but you do something else that addresses the underlying risk. Name what you do, and explain why it's equivalent:

> Q: Do you maintain a hot disaster recovery site?
> A: We use a multi-region active-passive architecture with continuous data replication and a tested RTO of 4 hours and RPO of 15 minutes. We do not maintain a separate "DR site" in the traditional sense; the recovery target is a cold-but-rehearsable deployment to our secondary AWS region. Evidence: DR runbook, last DR drill report (March 2026).

The compensating-control answer is where SMB responses earn trust. CISOs know smaller companies make architectural choices different from enterprises. What they want to know is that you made the choice deliberately and understand its risk profile.

### Pattern 4: Not applicable

The question is about a capability you don't have because your architecture doesn't have the surface area. State that, and briefly explain why:

> Q: Describe your data-in-transit encryption for SCADA/OT systems.
> A: Not applicable. Our platform does not operate or integrate with industrial control systems or operational technology environments.

Don't try to answer NA questions. Don't write "see other answer." A clear "not applicable, here's why" gets the auditor moving on. A vague "we follow industry best practices" gets you flagged for a follow-up call.

## Common Frameworks You'll See

Most questionnaires draw from one of a few standard frameworks. Recognizing them saves time:

- **SIG (Shared Assessments Standard Information Gathering)** — the most common enterprise questionnaire. SIG-Lite is ~100 questions; full SIG is ~1,000. Most prospects send SIG-Lite or a subset.
- **CAIQ (Consensus Assessments Initiative Questionnaire)** — cloud-specific, ~300 questions, maintained by the Cloud Security Alliance.
- **VSAQ (Vendor Security Alliance Questionnaire)** — newer, more concise, less common.
- **Custom enterprise questionnaire** — many large enterprises maintain their own, usually borrowing 50-70% from SIG.

Recognizing the framework lets you leverage prior responses. If you've answered SIG-Lite before, your previous answers cover most of the next SIG-Lite. The first questionnaire is the expensive one; every subsequent one is incremental.

The single biggest shortcut, though, is a finished attestation: once you can attach a [SOC 2 report](/library/soc2-for-startups-complete-guide), whole blocks of the questionnaire collapse into a one-line "Yes — see report, section X."

## Implementation Notes

**Build a response repository.** Every answer you write goes into a versioned document keyed by question. The next time the same question appears (and it will), you copy-paste with light editing. Most teams maintain this in Notion, Confluence, or a dedicated GRC tool. The exact tool doesn't matter; the discipline does.

**Track answer-aging.** Answers go stale. The MFA coverage you reported six months ago might be different today. Tag every answer with a "last verified" date. Anything older than six months gets re-verified before reuse.

**Single owner per response.** Group-edited security questionnaires produce inconsistent answers. One person owns the document, with subject-matter reviewers for specific sections. The owner makes the consistency calls.

**Don't lie. Don't oversell.** Both poison the relationship and both come back to haunt you. The CISO will find out: follow-up clarification, contract renewal, or (worst case) a breach investigation. The questionnaire is the start of a trust relationship, not a hurdle to clear.

## The Template

Below is the response template for thirty of the most common questionnaire questions, with answer patterns. Adapt the specifics; use the structure.

| # | Question | Answer pattern |
|---|---|---|
| 1 | Do you have a written information security policy? | Yes. Reviewed annually. Last review: [date]. |
| 2 | Do you have a designated CISO or security lead? | [Yes — name + title] OR [Compensating: head of platform engineering, security responsibilities documented in role description] |
| 3 | Describe your background check process for employees. | [Yes — vendor + scope] OR [Compensating: smaller team, manager-led reference checks, documented] |
| 4 | Do you provide security awareness training? | Yes. Annual training via [vendor], plus quarterly phishing simulations. |
| 5 | Do you encrypt data in transit? | Yes. TLS 1.2 minimum on all external endpoints; TLS 1.3 preferred. Internal traffic via [mesh / VPC isolation]. |
| 6 | Do you encrypt data at rest? | Yes. AES-256 via [KMS provider]. Customer-managed keys available on request. |
| 7 | Do you maintain an asset inventory? | Yes. [Tool] reconciled [frequency]. |
| 8 | Do you have a vulnerability management program? | Yes. [Scanner] running [frequency]; SLO of [X days] for critical, [Y] for high. |
| 9 | Do you perform penetration testing? | Yes. Annual third-party pen test via [vendor]. Last test: [date]. Findings remediated. |
| 10 | Do you have a SOC2 / ISO 27001 / HITRUST attestation? | [Yes — current state] OR [In progress — target date] OR [Not currently certified — compensating: SOC 2 readiness assessment, evidence available on request] |
| 11 | How do you manage privileged access? | SSO-integrated, MFA-required, [JIT / standing], audit-logged. Privileged access reviewed quarterly. |
| 12 | Do you have logging and monitoring? | Yes. [Stack]. [Retention period]. Alerts route to 24/7 on-call. |
| 13 | Describe your incident response process. | Documented IR playbook. Severity 1 = page within 5 min, comms within 30 min. Last tabletop: [date]. |
| 14 | Do you maintain backups? | Yes. [Frequency, retention, cross-region status]. Restore drill annual; last: [date]. |
| 15 | Describe your DR posture. | RTO: [X hours]. RPO: [Y minutes]. Tested [frequency]. |
| 16 | Do you have a BCP? | Yes. Reviewed [frequency]. |
| 17 | How do you handle subcontractors/sub-processors? | Maintained list. Annual security review. DPAs in place. |
| 18 | Do you support SSO via SAML/OIDC? | Yes. SAML 2.0 + OIDC. SCIM provisioning available on [tier]. |
| 19 | Do you support customer-managed encryption keys? | [Yes — how] OR [Compensating: AWS KMS-managed with customer-specific key per tenant] |
| 20 | What's your data residency? | [Region(s)]. Data does not leave [region] without [process]. |
| 21 | How do you handle data deletion / right-to-be-forgotten? | Documented process; X-day SLO. Soft-delete + hard-delete distinction. |
| 22 | Are you HIPAA-compliant? | [BAA available; HIPAA-aware architecture; supporting controls documented] |
| 23 | Are you GDPR-compliant? | [DPA available; sub-processor list maintained; DPO designated] |
| 24 | Are you PCI compliant? | [Yes — level] OR [Not applicable — we don't process card data, payments go via [provider]] |
| 25 | How long is your data retention? | [Policy + tier-specific retention] |
| 26 | Do you maintain change management? | Yes. Every production change tied to a PR; security-relevant changes require additional review. Audit log retained [period]. |
| 27 | Do you have an exception process? | Yes. Documented. Approvals tracked. Reviewed quarterly. |
| 28 | How do you handle EOL/EOS dependencies? | Automated CVE scanning. SLO of [X] for critical CVEs. Quarterly stack review. |
| 29 | Do you have a security champions program? | [Yes — structure] OR [Compensating: security review on every PR; designated security reviewer in engineering] |
| 30 | Where can I find your trust portal / security page? | Link the trust page you actually keep current. Do not send a reviewer to a page that no longer matches how you operate. |

The thirty questions above cover ~70% of any typical questionnaire. The other 30% are environment-specific (your specific tech stack, your specific data flows). With the thirty in place as a baseline, the environment-specific questions become the focused work.

## Operating Notes

Treat the questionnaire as a strategic asset, not a tactical inconvenience. The answers tell prospects what you stand for operationally. Brief, honest, specific answers signal that you take security seriously. Hedge-filled, marketing-flavored answers signal that you don't.

For the first three or four questionnaires your team handles, the founder or CTO should be in the response loop. After that, the response repository is mature enough that a security or platform engineer can run the process end-to-end with senior review only on the trickier questions.

The first questionnaire takes a week. The fifth takes a day. The fiftieth is a copy-paste plus a delta review.

## Frequently asked questions

**Q: How long should it take to answer an enterprise security questionnaire?**

A team that has been through it a couple of times can work through even a 240-question questionnaire in about four hours and ship a response that closes the deal without a clarification call. The first questionnaire takes roughly a week; by the fifth it is down to a day, and the fiftieth is a copy-paste plus a delta review.

**Q: What are enterprise security questionnaires actually testing?**

Primarily whether you've thought about each control well enough to give an honest answer, not whether you've implemented every control perfectly. A CISO doesn't expect a Series A company to have a Fortune 500's posture; they want you to know your posture, describe it accurately, and explain what is compensating where coverage is incomplete.

**Q: How do I answer when we don't have the exact control they're asking about?**

Use a compensating control: name what you do instead and explain why it addresses the same underlying risk. If the capability doesn't apply to your architecture at all, mark it 'not applicable' with a brief reason rather than writing 'see other answer' or 'we follow industry best practices,' which gets you flagged for a follow-up call.

**Q: What security questionnaire frameworks will I run into?**

The most common is SIG (Shared Assessments): SIG-Lite is about 100 questions and full SIG about 1,000. CAIQ is cloud-specific at roughly 300 questions, VSAQ is newer and more concise, and many large enterprises use a custom questionnaire that borrows 50-70% from SIG. Recognizing the framework lets you reuse prior answers.

**Q: Should one person or the whole team fill out the questionnaire?**

One person should own the document and make the consistency calls, with subject-matter reviewers for specific sections, because group-edited questionnaires produce inconsistent answers. Keep the founder or CTO in the loop for your first three or four; after that a security or platform engineer can run it end-to-end with senior review only on the tricky questions.

**Q: Does a SOC 2 report help with security questionnaires?**

Yes, and a finished attestation is the single biggest shortcut. Once you can attach a SOC 2 report, whole blocks of the questionnaire collapse into a one-line 'Yes, see report, section X.'

Template: "Pre-filled response template for 30 common questions" — branded PDF at https://kaansystems.com/templates/reading-security-questionnaire.pdf

Source: https://kaansystems.com/library/reading-security-questionnaire · Work with Kaan Systems: https://kaansystems.com/contact

---

# Securing PHI in Observability Data

Canonical: https://kaansystems.com/library/securing-phi-observability
Topic: Compliance · Published: 2026-05-12 · Updated: 2026-10-12
Summary: Modern observability pipelines collect everything. That's the value, and it's also the problem. PHI leaks into logs, traces, and metrics through routes most teams don't audit. Defense in depth, layer by layer.

## Key takeaways

- PHI leaks into observability data through routes most teams never audit: URL parameters, verbose application logs, database query strings, error stack traces, and distributed trace spans.
- Any PHI sitting in your monitoring stack is almost certainly outside the HIPAA scope you architected for production data stores, which makes it a breach.
- Defense in depth means four layers, each cheaper to fix than the next one down: don't generate the PHI, scrub it at the ingestion pipeline, lock down storage, and treat tracing as its own special case.
- Scrub aggressively at the pipeline layer. False positives are nearly free in observability data because the real value still lives in your application database, where it belongs.
- Lock down the observability store like a production database: encrypt at rest, gate access with SSO plus MFA plus a scoped role, turn on the query audit log, and keep hot retention to about 30 days.
- Skipping tracing entirely for PHI-touching services is a reasonable trade against breach risk, and pattern-based scrubbing will never catch a structured leak like an extra={'patient_id': ...} field — only code review will.

**How do you keep PHI out of your observability data?** Defense in depth across four layers: don't generate the PHI at the application, scrub what leaks through at the ingestion pipeline, lock down the storage that still catches some, and treat tracing as the special case it is. Each layer is cheaper to fix than the one below it.

The scrub belongs on the same path as an [evidence pipeline](/library/evidence-pipelines-101). The account split that keeps logs away from the workload is [minimum-viable HIPAA on AWS](/library/minimum-viable-hipaa-aws). If prompts are a second PHI store, [LLM feature auditability](/library/llm-feature-auditability) is the trace shape that does not keep the content.

## How does PHI end up in logs and traces?

Observability stacks exist to capture as much as possible about what your systems are doing. Datadog, New Relic, Honeycomb, an in-house ELK setup, doesn't matter. That's their value. It also makes them PHI minefields in any healthcare-adjacent environment.

The leaks happen through routes most teams don't audit:

- **Request URL parameters.** `/api/patients?ssn=...` writes the SSN into every access log and trace span.
- **Verbose application logs.** "Loaded user 12345 (John Smith, DOB 1980-04-12) from RDS." Written during an incident two years ago, never removed, indexed in Elasticsearch ever since.
- **Database query strings.** Slow-query logs that include the query text capture every WHERE clause that touched PHI columns.
- **Error stack traces.** An unhandled exception in a function processing patient data often captures the entire request body in the error message.
- **Distributed traces.** A trace span across services naturally includes the entity IDs and frequently the entity contents at each hop.

Any of these put protected health information into a system that's almost certainly outside the HIPAA scope you've architected for production data stores. That's a breach.

## How do you keep observability out of a breach?

Defense in depth. Four layers, each cheaper to fix than the next one down:

```mermaid
flowchart TB
  subgraph APP[Layer 1 · Application — don't generate it]
    direction LR
    A1[no_log_body decorator]
    A2[Opaque IDs in URLs]
    A3[before_send hooks<br/>strip PHI from errors]
    A4[DB query logger<br/>no parameter values]
  end

  subgraph PIPE[Layer 2 · Pipeline — scrub at ingestion]
    direction LR
    B1[SSN · MRN · DOB regex]
    B2[Email · phone · CC regex]
    B3[Bearer · API-key regex]
  end

  subgraph STORE[Layer 3 · Storage — contain the blast radius]
    direction LR
    C1[Encrypt at rest · CMKs]
    C2[SSO + MFA + scoped role]
    C3[Query audit log on]
    C4[Hot retention ≤ 30d]
  end

  subgraph TRACE[Layer 4 · Tracing — treat specially]
    direction LR
    D1[Sampling exclusion]
    D2[Span attribute filter]
    D3[Service-level data class tag]
  end

  APP --> PIPE --> STORE --> TRACE
  TRACE --> SAFE((PHI<br/>contained))

  classDef l1   fill:#3FA585,stroke:#3FA585,stroke-width:2px,color:#0a1612;
  classDef l2   fill:#2E8B6F,stroke:#3FA585,stroke-width:2px,color:#0a1612;
  classDef l3   fill:#1F4A3E,stroke:#3FA585,stroke-width:2px,color:#E8F5EE;
  classDef l4   fill:#0d1f1a,stroke:#3FA585,stroke-width:1.5px,color:#E8F5EE;
  classDef safe fill:#0a1612,stroke:#3FA585,stroke-width:2.5px,color:#E8F5EE;
  classDef band fill:#2E8B6F0A,stroke:#2E8B6F,stroke-width:1px;

  class A1,A2,A3,A4 l1;
  class B1,B2,B3 l2;
  class C1,C2,C3,C4 l3;
  class D1,D2,D3 l4;
  class SAFE safe;
  class APP,PIPE,STORE,TRACE band;
```

### Layer 1: Don't generate it

The cheapest log to scrub is the one you never wrote. Engineering hygiene at the source:

- **Never log full request bodies for PHI-bearing endpoints.** Log the request shape (method, path, status, latency) but not the payload. Endpoints that touch PHI should have a `@no_log_body` decorator or equivalent enforced.
- **Use opaque identifiers in URL paths, not PHI.** `/api/patients/PAT-7891234` is fine. `/api/patients/?ssn=...` is not. Most teams need an internal-ID layer for this.
- **Sanitize error contexts.** Wrap every PHI-touching function in an exception handler that strips PHI from the stack trace before it propagates. Most observability SDKs support `before_send` hooks for exactly this.
- **Set the database query logger to not include parameter values.** Most ORMs default to including parameters in slow-query logs. Override that.

### Layer 2: Scrub at the pipeline

For everything you couldn't prevent at the source, the next defense is at the ingestion layer.

Every major observability vendor supports a "pre-send" or "scrubber" filter that runs against payloads before they leave the agent. Datadog calls it a Sensitive Data Scanner; Splunk has CIM-based redaction; Elastic has ingest pipeline processors; New Relic has attribute filters.

The patterns to scrub are well-known. The template below covers the common ones (SSN, MRN, DOB, email, phone, credit card). Configure them once at the agent level so they apply across every service.

The common mistake here is scrubbing too narrowly. Scrub *aggressively*. False positives are nearly free in observability data. A log line with "[REDACTED]" where a value used to be is fine for debugging context; the actual value is in your application database, where it belongs.

### Layer 3: Lock down storage

Even with scrubbing, some PHI inevitably leaks. The storage layer assumes that and contains the blast radius:

- **Encrypt the observability data store at rest.** With customer-managed keys if you're large enough to care about key access audit.
- **Restrict access via SSO + MFA + an explicit access role.** Engineers reach for observability tools more often than production databases; the access bar should be similar.
- **Audit who queries what.** Most observability vendors offer an audit log of queries run; turn it on. The next incident that involves "someone accessed PHI in logs," that audit log is your evidence.
- **Set short retention on raw logs.** Hot retention (queryable, fast) of 30 days for application logs is enough for ~95% of debug cases. Anything older goes to cold archive with stricter access.

### Layer 4: Treat tracing carefully

Traces are the hardest case. A trace span captures the context flowing across services, and in a microservices environment touching PHI, the "context" naturally includes patient identifiers, sometimes more.

Three patterns help:

- **Sampling exclusion.** Configure the tracer to not sample traces from PHI-touching services for the storage layer. They still emit metrics; they just don't generate trace data that lives in the trace store.
- **Span attribute filtering.** Every major tracer supports attribute filtering before send. Strip any span attribute matching a PHI pattern.
- **Service-level data classification.** Tag services as PHI-touching at deploy time. The agent reads the tag and applies stricter filters to traces from those services.

A reasonable choice is to skip tracing entirely for PHI-touching services. The observability cost is real, but it's an acceptable trade-off against the breach risk.

## The Template

A scrubber config with regex patterns for the most common PHI types. Apply at the observability-agent layer (format shown for Datadog's Sensitive Data Scanner; patterns are vendor-independent).

| Field | Pattern | Replacement |
|---|---|---|
| SSN | `\b\d{3}-\d{2}-\d{4}\b` | `[REDACTED-SSN]` |
| SSN (no dashes) | `\b\d{9}\b` (context-dependent) | `[REDACTED-SSN]` |
| MRN (common formats) | `\b(MRN[-:\s]*\d{6,10}|\d{8})\b` | `[REDACTED-MRN]` |
| DOB | `\b(0[1-9]|1[012])[-/](0[1-9]|[12]\d|3[01])[-/](19|20)\d\d\b` | `[REDACTED-DOB]` |
| Email | `\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z\|a-z]{2,}\b` | `[REDACTED-EMAIL]` |
| Phone | `\b(\+\d{1,2}\s?)?\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4}\b` | `[REDACTED-PHONE]` |
| Credit card | `\b(?:\d{4}[-\s]?){3}\d{4}\b` | `[REDACTED-CC]` |
| Auth bearer token | `Bearer\s+[A-Za-z0-9._~+/=-]+` | `Bearer [REDACTED]` |
| API key (generic) | `(sk|api)[-_][A-Za-z0-9]{20,}` | `[REDACTED-KEY]` |

For Datadog specifically, this lives at `Logs → Configuration → Pipelines → Sensitive Data Scanner`. The same patterns apply (sometimes with vendor-specific escape rules) to:

- **Elastic ingest pipelines** (`gsub` processors)
- **New Relic log obfuscation rules**
- **Splunk SEDCMD transforms**
- **Cloudflare Logpush filters**

Test the patterns against your actual logs before turning them on in production. Start in "tag-only" mode where matches are flagged but not modified, then flip to "replace" once you've verified no false-positive damage to debugging.

## Operating Notes

Add an audit step to your engineering checklist for new PHI-touching code: before merge, search the diff for likely log-leak candidates. A regex for `logger.info` or `console.log` followed by a variable reference catches most of them.

You will have leaks. Recovery is to immediately rotate the scrubber pattern to cover the new shape, then run the storage-layer access audit for the period the leak was live. If no one accessed the affected logs, you have a finding to document. If someone did, you have a more serious finding to escalate.

The hardest leak to catch is the structured one: a developer adds an `extra={'patient_id': ...}` field to a log call intending it to be opaque, but the field naming convention propagates patient identifiers into the index. Pattern-based scrubbing won't catch that. Only code review will.

## Frequently asked questions

**Q: How does PHI end up in logs and traces?**

It leaks in through routes most teams don't audit: SSNs and other identifiers in request URL parameters, verbose application logs that print names and dates of birth, database slow-query strings, request bodies captured in error stack traces, and entity data carried across distributed trace spans.

**Q: Is PHI in my monitoring stack actually a HIPAA breach?**

Yes. Your observability system is almost certainly outside the HIPAA scope you architected for production data stores, so any protected health information that lands there is PHI in an unauthorized system — a breach.

**Q: How do I keep PHI out of my observability data?**

Use defense in depth across four layers: don't generate the PHI at the application (no full request bodies, opaque IDs in URLs, sanitized error contexts), scrub what leaks at the ingestion pipeline, lock down the storage layer, and handle tracing specially. The cheapest log to scrub is the one you never wrote.

**Q: Should I scrub PHI patterns narrowly or aggressively?**

Aggressively. False positives are nearly free in observability data — a log line showing [REDACTED] where a value used to be is fine for debugging, because the actual value still lives in your application database. Scrubbing too narrowly is the common mistake.

**Q: How long should I keep observability logs that might contain PHI?**

Keep hot, queryable retention short — around 30 days for application logs covers roughly 95% of debug cases. Anything older should move to a cold archive with stricter access.

**Q: Should I trace services that handle PHI?**

Tracing is the hardest case because span context naturally carries patient identifiers. Options are sampling exclusion, span attribute filtering, and service-level data classification — but a reasonable choice is to skip tracing entirely for PHI-touching services, accepting the observability cost against the breach risk.

Template: "Log-scrubber regex config" — branded PDF at https://kaansystems.com/templates/securing-phi-observability.pdf

Source: https://kaansystems.com/library/securing-phi-observability · Work with Kaan Systems: https://kaansystems.com/contact