The Problem
An attacker who wants into your AWS account does not need a zero-day. They need one of the several hundred static credentials your organization has minted over the years, and they only need it once. GitHub is scraped for AKIA prefixes continuously; a canary access key pushed to a public repo receives its first unauthorized API call in under ten minutes. The CircleCI breach in January 2023 worked this way at one remove: an engineer’s laptop was compromised, a session was hijacked, and every customer who had stored AWS keys, deploy tokens, or signing certificates in CircleCI got the worst email of the quarter. Rotate everything, now, and hope your rotation runbooks exist.
The property that makes static credentials the dominant real-world breach vector is simple: stolen does not mean expired. A short-lived credential lifted from a compromised pod is worth an hour to an attacker. A long-lived one is worth however many months pass before someone notices, and breach-cost studies have put mean identification time for credential-based intrusions at around 240 days for years running. The secret in your CI settings, the IAM user key sitting in a developer’s ~/.aws/credentials since 2021, the database password in a Helm values file: each one works until a human revokes it, and revocation requires the human to know it was taken.
For a regulated SMB the problem doubles. HIPAA and SOC 2 both push you toward credential rotation, so teams perform quarterly rotation theater: a spreadsheet, a painful afternoon, and a couple of production incidents per year caused by the rotation itself. The exercise mostly proves that the process is manual and feared. The better answer is structural. Stop storing credentials at all, wherever a workload, a pipeline, or a human can prove its identity and be issued a short-lived credential on demand. For roughly 90% of a typical AWS estate, they can.
The Approach
The organizing principle: replace possession of a secret with proof of identity at every boundary. AWS already operates the exchange desk (STS). Your job is to register the identity providers and delete the static keys. The trust flow for a Kubernetes workload looks like this; the CI and human flows are the same shape with a different token issuer.
The inventory: where static credentials hide
Counting comes first. aws iam generate-credential-report lists every IAM user access key with its age and last-used date; on a typical 30-engineer estate that report shows 40 to 80 keys, a third of them unused in the last 90 days. Then the places the report cannot see: repository and organization secrets in GitHub Actions, database passwords in ECS task definitions and Helm values, kubeconfigs with embedded client certificates that never expire, service-account JSON files, and the vendor API key living in a founder’s dotfiles because it predates the secrets manager. Run gitleaks or trufflehog across full git history, not just HEAD; the file someone deleted in 2023 is still in the object store. The output is a register: credential, system, owner, blast radius if leaked, last-used date. The row count is the KPI for the entire program, and every phase below ends by making it smaller.
Workloads: identity instead of storage
IRSA (IAM Roles for Service Accounts) is the established path on EKS. Each cluster ships an OIDC issuer; you register it in IAM once, then write a role trust policy that matches the token’s sub claim to a specific service account, such as system:serviceaccount:prod:payments-api. The pod mounts a projected, auto-rotated token, and the AWS SDK exchanges it via AssumeRoleWithWebIdentity with no application changes. Credentials live in process memory, last an hour, and refresh themselves. Nothing on disk, nothing in a Kubernetes Secret, nothing in etcd to exfiltrate.
EKS Pod Identity (GA since late 2023) reaches the same end with less ceremony: no per-cluster OIDC provider registration, an agent handling AssumeRoleForPodIdentity, and role associations managed through a plain EKS API call. That matters once you run three or four clusters and tire of trust policies referencing cluster-specific issuer URLs. The trade-offs are minor but real: Pod Identity does not cover Fargate pods, and cross-account access needs explicit role chaining where IRSA can trust directly. Pick one mechanism per cluster and stay consistent. On GKE the same design is called Workload Identity Federation; on AKS, Entra Workload ID. Same JWT, same exchange.
CI/CD: federation, not repository secrets
The AWS_ACCESS_KEY_ID stored in GitHub repository secrets is usually the highest-blast-radius static credential an SMB owns: it carries deploy-level permissions and is exposed to every workflow, third-party action, and maintainer account. GitHub Actions issues OIDC tokens natively. Register token.actions.githubusercontent.com as an identity provider in IAM and the pipeline assumes a role directly, so the next CircleCI-class vendor breach is someone else’s rotation emergency.
The trust policy conditions do the security work (full JSON in the template below): aud pinned to sts.amazonaws.com, and sub pinned to repo:your-org/your-repo:ref:refs/heads/main. The classic misconfiguration is a StringLike condition on repo:your-org/*, which lets any workflow in any repository in the org — including a pull request against some archived experiment — assume your deploy role. Security researchers have found repo:* in the wild, which is an open door for every GitHub account on the planet. Pin to repository and ref, and give pull-request builds a separate role that can run terraform plan and nothing else.
Humans: SSO sessions, zero IAM users
Put IAM Identity Center in front of whatever IdP you already pay for (Google Workspace, Okta, Entra). Engineers get permission sets, sessions of 8 to 12 hours, and aws sso login for the CLI. Then the uncomfortable part: delete the IAM users. All of them. The end state is a credential report with zero rows, plus two break-glass IAM users with hardware MFA whose access keys exist only on paper in a safe, wired to a CloudTrail alarm that pages on any use. This is Phase 1 of zero trust for SMBs applied to the AWS control plane: identity-based, short-lived, centrally revocable. When someone leaves, disabling one IdP account kills every session within hours. Compare that with the offboarding checklist line that reads “rotate any keys they may have had.”
Databases and third-party APIs
RDS IAM authentication swaps the password for a 15-minute connection token, on Postgres, MySQL, and MariaDB engines including Aurora. It fits services with modest connection churn and works cleanly behind RDS Proxy. It does not fit everything: IAM-auth connection establishment carries extra overhead on the database side — token verification and mandatory TLS — and AWS recommends it only below roughly 200 new connections per second; some ORMs fight the auth plugin, and per-user tokens map awkwardly onto shared connection pools without a proxy in the path.
Where a password must exist, make it self-rotating. Secrets Manager with the managed Lambda rotators handles RDS credentials on a schedule using the alternating-users strategy, and applications that fetch the secret at connect time never notice a rotation. Vault’s dynamic database secrets go further: per-lease database users created on demand with a TTL, giving every workload its own expiring identity inside the database. Honest guidance at 10-80 engineers: Vault is a distributed system you have to operate (HA, unseal, upgrades, policy sprawl), and running it solely for database credentials is overkill when Secrets Manager rotation delivers most of the value with none of the pager load. Vault earns its keep when you are multi-cloud, need dynamic credentials across many engines, or already run it for something else. HCP Vault splits the difference if lease-based credentials are genuinely required without the ops burden.
The migration: parallel run, then revoke
Four moves per credential class: inventory, build the new issuance path, run both in parallel, revoke the old one. Deactivation is the safety net. IAM access keys support an Inactive state, so flip a key inactive for a week before deleting it and watch CloudTrail for whatever screams. Last-used data sets the sequence: dead keys first (instant wins, zero risk), then CI, then workloads, then humans. Publish the static-credential count weekly. A graph that reads 74, 61, 40, 19 does more for program momentum than any policy document.
Then the residue: the three vendors that only support API keys. There are always three. Scope each key to the minimum the integration needs, store it in Secrets Manager, put a 90-day rotation cadence on the calendar, assign a named owner, and alert on usage anomalies. Those keys become line items in your vendor risk scoring, because their rotation story is now part of the vendor’s risk posture, not only yours.
The Template
The migration checklist below is phased so that each phase ends with a revocation, not a deployment. A phase is not done while the old credential still works.
# No-Long-Lived-Credentials Migration
## Phase 0: Inventory (week 1)
- [ ] Run `aws iam generate-credential-report`; export all access keys with age and last-used
- [ ] Enumerate CI secrets: GitHub org, repo, and environment secrets
- [ ] Grep deploy artifacts for embedded creds: task definitions, Helm values, compose files, .env
- [ ] gitleaks/trufflehog across full git history of all active repos
- [ ] List kubeconfigs with embedded client certs; note expiry (often: none)
- [ ] Build the register: credential | system | owner | blast radius | last used
- [ ] Record the baseline count; this number is the program KPI
## Phase 1: Workloads (weeks 2-4)
- [ ] Register cluster OIDC provider in IAM (IRSA) or enable the EKS Pod Identity agent
- [ ] One IAM role per service; trust pinned to namespace + service account
- [ ] Migrate services off node-role and env-var creds; verify AssumeRoleWithWebIdentity in CloudTrail
- [ ] Delete AWS credentials from Kubernetes Secrets and task-definition env vars
## Phase 2: CI/CD (weeks 3-5)
- [ ] Create the token.actions.githubusercontent.com OIDC provider in IAM
- [ ] Deploy role trust pinned to repo + branch (JSON below); separate plan-only role for PRs
- [ ] Cut workflows over to aws-actions/configure-aws-credentials with role-to-assume
- [ ] Deactivate, then delete, the CI IAM user and its stored repo secrets
## Phase 3: Humans (weeks 4-6)
- [ ] IAM Identity Center connected to the IdP; permission sets replace per-user policies
- [ ] CLI access via `aws sso login`; session duration 12h or less
- [ ] Create 2 break-glass IAM users: hardware MFA, keys stored offline, CloudTrail alarm on use
- [ ] Flip all remaining IAM user keys Inactive for 7 days, then delete the users
## Phase 4: Data stores (weeks 5-8)
- [ ] RDS IAM auth for low-churn services (Postgres/MySQL/Aurora)
- [ ] Secrets Manager rotation (managed Lambda, alternating users) for the rest
- [ ] Apps fetch DB credentials at connect time from Secrets Manager, never from env vars
- [ ] Rotate every DB password once through the new path to invalidate leaked history
## Phase 5: Third parties (ongoing)
- [ ] Per vendor key: minimum scope, Secrets Manager entry, named owner, 90-day rotation
- [ ] Usage alerting per key (API audit log or billing anomaly)
- [ ] Fold key-rotation posture into vendor risk reviews
## Steady state
- [ ] Static-credential count published weekly (target: break-glass plus vendor keys, nothing else)
- [ ] CI check blocking AKIA* and other known key formats in commits
- [ ] SCP denying iam:CreateUser and iam:CreateAccessKey outside the break-glass path (SCPs never bind the Organizations management account, so this holds only if workloads live in member accounts; single-account estates: explicit deny in a permissions boundary plus a CloudTrail/EventBridge alarm on CreateUser/CreateAccessKey)The GitHub Actions trust policy, with the conditions that matter:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::111122223333:oidc-provider/token.actions.githubusercontent.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub": "repo:your-org/your-repo:ref:refs/heads/main"
}
}
}
]
}Never widen the sub condition to repo:your-org/* on the deploy role. If several repos genuinely need deploy access, give each its own statement or its own role.
Operating Notes
Revocation is the real test
If revoking a particular credential feels frightening, you have located a system whose ownership and issuance path nobody understands. That fear is the finding; write it down before you fix it. The parallel-run pattern exists to convert fear into boredom: the new path serves traffic for a week while the old credential sits deactivated but recoverable. Apply the same discipline to break-glass. Twice a year, actually open the safe, actually use the offline key, and confirm the alarm pages someone. A break-glass procedure that has never been exercised is a decorative one.
The audit trail arrives for free
Every AssumeRole and AssumeRoleWithWebIdentity call is a CloudTrail event binding an action to an identity, a session, and a timestamp. Once credentials are issued rather than stored, your rotation control is satisfied structurally: the auditor question “how do you rotate credentials?” gets the answer “they expire hourly, and issuance is logged.” That is a materially stronger position for SOC 2 CC6.1 than a spreadsheet of quarterly rotations, and it feeds directly into evidence pipelines, because the proof is a queryable log rather than a screenshot someone remembers to take.
Don’t chase the last three
The static-credential count follows an asymptote. It falls fast through the dead keys and CI secrets, slower through data stores, and then lands on a small stubborn number: break-glass plus the vendors that will not do OIDC. Accept the asymptote and guard the perimeter instead. An SCP denying new access-key creation means the count cannot silently climb back up, and a CI secret-format check means the next leaked key is caught at commit time. The durable win is cultural: engineers who joined after the migration have never configured a static AWS key and would not know where to put one. At that point the migration has stopped being a project and become the default, which is the only finish line that holds.